2024-03-30T10:40:08.255 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T10:40:08.255 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473 --verbose --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} -- /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473/orig.config.yaml 2024-03-30T10:40:08.278 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T10:40:08.351 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473 branch: wip-ceph_test_rados-partial-reads description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7630473' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-ceph_test_rados-partial-reads ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc workunit: branch: wip-ceph_test_rados-partial-reads sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9805 sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-ceph_test_rados-partial-reads suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsWYmf6Xl2fk/Utdo9pthxdi5pqHBqbjubEp7sYVvzAuoBQLg/OOaI4+ZCiLim6+jcEg8UegHmnG+KiyYyQAmQ= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-30_00:41:04 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T10:40:08.351 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa; will attempt to use it 2024-03-30T10:40:08.351 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks 2024-03-30T10:40:08.352 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T10:40:08.353 INFO:teuthology.task.internal:Checking packages... 2024-03-30T10:40:08.376 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc' 2024-03-30T10:40:08.376 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T10:40:08.376 INFO:teuthology.packaging:ref: None 2024-03-30T10:40:08.376 INFO:teuthology.packaging:tag: None 2024-03-30T10:40:08.377 INFO:teuthology.packaging:branch: wip-ceph_test_rados-partial-reads 2024-03-30T10:40:08.377 INFO:teuthology.packaging:sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T10:40:08.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-ceph_test_rados-partial-reads 2024-03-30T10:40:08.530 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1864.g5dc2ea30 2024-03-30T10:40:08.531 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T10:40:08.537 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T10:40:08.537 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T10:40:08.561 INFO:teuthology.task.internal:Saving configuration 2024-03-30T10:40:08.570 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T10:40:08.581 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T10:40:08.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 10:35:24.815388', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsWYmf6Xl2fk/Utdo9pthxdi5pqHBqbjubEp7sYVvzAuoBQLg/OOaI4+ZCiLim6+jcEg8UegHmnG+KiyYyQAmQ='} 2024-03-30T10:40:08.604 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T10:40:08.610 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-30T10:40:08.610 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T10:40:08.645 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3764cedf70>, signals=[15]) 2024-03-30T10:40:08.645 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T10:40:08.651 INFO:teuthology.task.internal:Opening connections... 2024-03-30T10:40:08.651 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-03-30T10:40:08.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T10:40:08.757 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T10:40:08.764 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-03-30T10:40:08.827 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-03-30T10:40:08.827 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-03-30T10:40:08.884 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-03-30T10:40:08.884 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-03-30T10:40:08.884 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-03-30T10:40:08.884 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-03-30T10:40:08.884 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-03-30T10:40:08.884 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-30T10:40:08.885 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-30T10:40:08.886 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-03-30T10:40:08.907 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T10:40:08.916 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T10:40:08.921 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T10:40:08.922 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-03-30T10:40:08.940 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T10:40:08.947 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T10:40:08.947 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-03-30T10:40:09.011 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T10:40:09.046 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T10:40:09.060 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T10:40:09.060 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T10:40:09.061 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T10:40:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T10:40:09.077 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-30T10:40:09.135 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-03-30T10:40:09.135 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-03-30T10:40:09.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-03-30T10:40:11.289 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-30T10:40:12.772 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-30T10:40:14.021 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-03-30T10:40:18.623 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 19 MB 00:03 2024-03-30T10:41:37.247 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 84 kB/s | 6.1 MB 01:15 2024-03-30T10:41:38.938 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-30T10:41:39.973 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-30T10:41:45.040 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-30T10:41:47.380 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-30T10:41:47.381 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-30T10:41:47.479 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:41:47.487 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:41:47.487 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:41:47.487 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:41:47.487 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-03-30T10:41:47.488 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:41:47.489 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 76 M 2024-03-30T10:41:47.489 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 123 M 2024-03-30T10:41:47.489 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:41:47.642 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-30T10:41:48.209 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-30T10:41:48.725 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-30T10:41:49.059 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-30T10:41:49.062 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:41:49.062 INFO:teuthology.orchestra.run.smithi003.stdout:Total 49 MB/s | 76 MB 00:01 2024-03-30T10:41:49.062 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:41:49.503 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:41:49.503 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:41:50.380 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:41:50.380 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:41:52.805 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:41:53.416 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-30T10:41:53.511 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T10:41:56.400 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T10:41:56.529 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T10:42:01.636 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T10:42:01.678 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T10:42:06.800 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T10:42:37.862 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T10:42:37.863 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T10:42:37.863 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:42:37.906 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-30T10:42:38.308 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-30T10:42:38.308 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-30T10:42:38.308 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-30T10:42:38.308 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-30T10:42:38.732 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-30T10:42:38.732 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:42:38.732 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:42:38.732 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-30T10:42:38.733 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-30T10:42:38.733 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-30T10:42:38.733 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-30T10:42:38.733 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:42:38.733 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:42:39.078 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T10:42:39.178 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T10:42:39.179 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T10:42:39.179 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-30T10:42:39.180 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T10:42:39.180 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T10:42:39.180 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T10:42:39.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T10:42:39.180 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-30T10:42:39.264 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-30T10:42:39.264 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-30T10:42:39.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-03-30T10:42:39.302 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-03-30T10:42:39.303 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:42:39.304 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T10:42:39.305 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T10:42:39.305 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-434.el9.x86_64 2024-03-30T10:42:39.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T10:42:39.975 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-03-30T10:42:41.330 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-30T10:42:41.364 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-03-30T10:42:41.366 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-30T10:42:41.382 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.xRv7c6WVmi 2024-03-30T10:42:41.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xRv7c6WVmi 2024-03-30T10:42:41.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.xRv7c6WVmi 2024-03-30T10:42:41.602 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.xRv7c6WVmi is 6KB 2024-03-30T10:42:41.652 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.xRv7c6WVmi 2024-03-30T10:42:41.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T10:42:41.736 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-30T10:42:41.736 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-30T10:42:41.736 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-30T10:42:41.736 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-30T10:42:41.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-30T10:42:41.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-03-30T10:43:11.926 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T10:43:11.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-30T10:43:11.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T10:43:30.463 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-03-30T10:43:39.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-30T10:43:39.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T10:44:39.535 DEBUG:teuthology.orchestra.remote:timed out 2024-03-30T10:44:51.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-30T10:44:51.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T10:44:51.805 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-30T10:44:52.181 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-30T10:44:52.181 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-30T10:44:52.181 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-30T10:44:52.197 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-434.el9.x86_64 2024-03-30T10:44:52.197 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-30T10:44:52.197 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T10:44:52.197 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-30T10:44:53.198 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T10:44:53.199 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T10:44:53.262 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-03-30T10:44:53.294 DEBUG:teuthology.parallel:result is None 2024-03-30T10:44:53.294 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T10:44:53.302 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T10:44:53.302 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T10:44:53.320 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T10:44:53.330 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T10:44:53.342 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T10:44:53.342 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T10:44:53.390 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T10:44:53.400 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T10:44:53.401 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T10:44:53.466 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:44:53.480 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:44:53.482 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T10:44:53.489 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T10:44:53.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T10:44:53.554 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T10:44:53.563 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T10:44:53.563 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T10:44:53.610 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T10:44:53.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T10:44:53.752 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T10:44:53.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T10:44:53.838 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:44:53.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T10:44:53.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-03-30T10:44:53.987 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T10:44:54.172 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T10:44:54.181 INFO:teuthology.task.internal:Starting timer... 2024-03-30T10:44:54.181 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T10:44:54.191 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T10:44:54.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-03-30T10:44:54.267 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-03-30T10:44:54.269 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T10:44:54.269 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-03-30T10:44:54.292 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-03-30T10:44:54.292 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive'} 2024-03-30T10:44:54.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T10:44:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T10:44:54.360 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T10:44:54.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-03-30T10:44:54.436 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T10:44:54.447 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T10:44:54.467 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T10:44:54.469 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T10:49:12.797 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com')] 2024-03-30T10:49:12.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-30T10:49:12.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T10:49:12.895 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-30T10:49:12.973 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-30T10:49:12.973 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T10:49:12.984 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T10:49:12.984 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T10:49:12.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T10:49:13.045 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T10:49:13.063 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T10:49:13.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-03-30T10:49:13.106 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-03-30T10:49:13.124 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T10:49:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T10:49:13.189 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-30T10:49:13.193 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T10:49:13.193 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-30T10:49:13.193 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T10:49:13.193 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T10:49:13.193 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T10:49:13.193 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T10:49:13.194 INFO:teuthology.run_tasks:Running task install... 2024-03-30T10:49:13.202 DEBUG:teuthology.task.install:project ceph 2024-03-30T10:49:13.203 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}} 2024-03-30T10:49:13.203 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'} 2024-03-30T10:49:13.203 INFO:teuthology.task.install:Using flavor: default 2024-03-30T10:49:13.208 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T10:49:13.209 INFO:teuthology.task.install:extra packages: [] 2024-03-30T10:49:13.209 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'tag': None, 'wait_for_package': False} 2024-03-30T10:49:13.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T10:49:13.365 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/centos/9/flavors/default/ 2024-03-30T10:49:13.365 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1864.g5dc2ea30 2024-03-30T10:49:13.539 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T10:49:13.539 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:49:13.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T10:49:13.567 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-30T10:49:13.567 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T10:49:13.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T10:49:13.716 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T10:49:13.782 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-03-30T10:49:13.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-03-30T10:49:14.038 INFO:teuthology.orchestra.run.smithi003.stdout:74 files removed 2024-03-30T10:49:14.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2024-03-30T10:49:14.774 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 204 kB/s | 82 kB 00:00 2024-03-30T10:49:15.191 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-30T10:49:15.600 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-30T10:49:15.994 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-03-30T10:49:17.444 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-30T10:49:19.427 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-03-30T10:49:23.636 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-03-30T10:49:35.277 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 793 kB/s | 6.1 MB 00:07 2024-03-30T10:49:36.995 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-30T10:49:38.907 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-30T10:49:43.955 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-30T10:49:46.642 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:49:46.654 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:49:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:49:46.654 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:49:46.654 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:49:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 9.8 M 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 3.3 M 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 3.1 M 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 5.4 M 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 20 M 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 25 k 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-30T10:49:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 745 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 501 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 5.1 M 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 45 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 131 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 162 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-30T10:49:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 320 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 298 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 99 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:49:46.657 INFO:teuthology.orchestra.run.smithi003.stdout:Install 30 Packages 2024-03-30T10:49:46.658 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-03-30T10:49:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:49:46.658 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 58 M 2024-03-30T10:49:46.658 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:49:47.331 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-base-19.0.0-1864.g5dc2ea30.el9.x86 8.7 MB/s | 5.4 MB 00:00 2024-03-30T10:49:47.389 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-selinux-19.0.0-1864.g5dc2ea30.el9. 431 kB/s | 25 kB 00:00 2024-03-30T10:49:47.490 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): libcephfs2-19.0.0-1864.g5dc2ea30.el9.x8 7.3 MB/s | 745 kB 00:00 2024-03-30T10:49:47.573 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): libradosstriper1-19.0.0-1864.g5dc2ea30. 5.9 MB/s | 501 kB 00:00 2024-03-30T10:49:48.032 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): librgw2-19.0.0-1864.g5dc2ea30.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-30T10:49:48.090 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): python3-ceph-argparse-19.0.0-1864.g5dc2 776 kB/s | 45 kB 00:00 2024-03-30T10:49:48.149 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): python3-ceph-common-19.0.0-1864.g5dc2ea 2.2 MB/s | 131 kB 00:00 2024-03-30T10:49:48.358 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): ceph-common-19.0.0-1864.g5dc2ea30.el9.x 12 MB/s | 20 MB 00:01 2024-03-30T10:49:48.383 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): python3-cephfs-19.0.0-1864.g5dc2ea30.el 693 kB/s | 162 kB 00:00 2024-03-30T10:49:48.408 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): python3-rados-19.0.0-1864.g5dc2ea30.el 6.3 MB/s | 320 kB 00:00 2024-03-30T10:49:48.442 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): python3-rbd-19.0.0-1864.g5dc2ea30.el9. 5.0 MB/s | 298 kB 00:00 2024-03-30T10:49:48.467 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): python3-rgw-19.0.0-1864.g5dc2ea30.el9. 1.7 MB/s | 99 kB 00:00 2024-03-30T10:49:48.592 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-03-30T10:49:48.618 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-30T10:49:48.643 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-30T10:49:48.702 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-30T10:49:48.727 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-30T10:49:48.752 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-03-30T10:49:48.786 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-30T10:49:48.811 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 597 kB/s | 35 kB 00:00 2024-03-30T10:49:48.837 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-30T10:49:49.245 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 712 kB/s | 309 kB 00:00 2024-03-30T10:49:49.312 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-30T10:49:49.396 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 584 kB/s | 49 kB 00:00 2024-03-30T10:49:49.563 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 403 kB/s | 67 kB 00:00 2024-03-30T10:49:49.638 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-03-30T10:49:49.714 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-30T10:49:49.747 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-30T10:49:49.856 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): ceph-radosgw-19.0.0-1864.g5dc2ea30.el9 3.1 MB/s | 9.8 MB 00:03 2024-03-30T10:49:49.906 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-03-30T10:49:50.065 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): librados2-19.0.0-1864.g5dc2ea30.el9.x8 11 MB/s | 3.3 MB 00:00 2024-03-30T10:49:50.232 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): librbd1-19.0.0-1864.g5dc2ea30.el9.x86_ 8.4 MB/s | 3.1 MB 00:00 2024-03-30T10:49:50.238 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:49:50.238 INFO:teuthology.orchestra.run.smithi003.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-30T10:49:50.239 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:49:50.281 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:49:50.281 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:49:50.782 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:49:50.783 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:49:52.287 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:49:52.380 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-30T10:49:52.417 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-30T10:49:52.446 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-30T10:49:52.476 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T10:49:52.493 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-30T10:49:52.685 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T10:49:52.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-30T10:49:53.096 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-30T10:49:53.155 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 7/34 2024-03-30T10:49:53.249 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 7/34 2024-03-30T10:49:53.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:19.0.0-1864.g5dc2ea30.el9.x86_64 8/34 2024-03-30T10:49:53.382 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 9/34 2024-03-30T10:49:53.445 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 9/34 2024-03-30T10:49:53.473 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-30T10:49:53.517 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:19.0.0-1864.g5dc2ea30.el9. 11/34 2024-03-30T10:49:53.588 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:19.0.0-1864.g5dc2ea30.el9.x86_64 12/34 2024-03-30T10:49:53.609 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:19.0.0-1864.g5dc2ea30.el9.x86_6 13/34 2024-03-30T10:49:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1864.g5dc2ea30.el9.x86_6 13/34 2024-03-30T10:49:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-30T10:49:53.836 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-30T10:49:53.867 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-30T10:49:53.909 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-30T10:49:53.949 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-30T10:49:53.986 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-30T10:49:54.027 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-30T10:49:54.356 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-30T10:49:54.391 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:19.0.0-1864.g5dc2ea30.el9.x86_64 22/34 2024-03-30T10:49:54.481 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:19.0.0-1864.g5dc2ea30.el9.x86_64 22/34 2024-03-30T10:49:54.511 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 23/34 2024-03-30T10:49:54.556 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-30T10:49:55.035 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-30T10:49:55.059 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 26/34 2024-03-30T10:49:55.145 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 26/34 2024-03-30T10:49:55.218 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 27/34 2024-03-30T10:49:55.254 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x8 28/34 2024-03-30T10:49:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x86_64 29/34 2024-03-30T10:49:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x86_64 29/34 2024-03-30T10:49:58.125 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x86_64 29/34 2024-03-30T10:49:58.163 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:19.0.0-1864.g5dc2ea30.el9.x86_64 30/34 2024-03-30T10:49:58.254 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:19.0.0-1864.g5dc2ea30.el9.x86_64 30/34 2024-03-30T10:49:58.254 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-30T10:49:58.254 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-30T10:49:58.255 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:49:58.289 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:19.0.0-1864.g5dc2ea30.el9.x86_64 31/34 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1864.g5dc2ea30.el9.x86_64 31/34 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-03-30T10:50:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:09.406 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 32/34 2024-03-30T10:50:09.430 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 32/34 2024-03-30T10:50:09.431 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:50:09.431 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-30T10:50:09.431 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T10:50:09.432 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-30T10:50:09.432 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:09.433 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T10:50:09.517 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-30T10:50:09.517 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T10:50:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 3/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:19.0.0-1864.g5dc2ea30.el9.x86_64 4/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 5/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:19.0.0-1864.g5dc2ea30.el9.x86_6 6/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 7/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1864.g5dc2ea30.el9. 8/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x8 9/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:19.0.0-1864.g5dc2ea30.el9.x86_64 10/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:19.0.0-1864.g5dc2ea30.el9.x86_64 11/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 12/34 2024-03-30T10:50:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 13/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-30T10:50:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-30T10:50:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-30T10:50:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-30T10:50:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 31/34 2024-03-30T10:50:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-30T10:50:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:19.0.0-1864.g5dc2ea30.el9.x86_64 33/34 2024-03-30T10:50:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-30T10:50:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:11.784 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-03-30T10:50:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-30T10:50:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-30T10:50:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-30T10:50:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2024-03-30T10:50:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-30T10:50:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:11.787 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:12.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2024-03-30T10:50:12.719 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:29 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:12.879 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:12.879 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 47 M 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 163 k 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-30T10:50:12.880 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:12.881 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 48 M 2024-03-30T10:50:12.881 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 205 M 2024-03-30T10:50:12.881 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:50:13.304 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-30T10:50:13.346 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-30T10:50:13.379 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-30T10:50:13.430 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): libcephsqlite-19.0.0-1864.g5dc2ea30.el9. 566 kB/s | 163 kB 00:00 2024-03-30T10:50:13.596 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 863 kB/s | 187 kB 00:00 2024-03-30T10:50:15.463 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-19.0.0-1864.g5dc2ea30.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-30T10:50:15.467 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:50:15.467 INFO:teuthology.orchestra.run.smithi003.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-30T10:50:15.482 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:50:15.505 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:50:15.506 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:50:15.908 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:50:15.908 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:50:16.492 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:50:16.565 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-30T10:50:16.603 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-30T10:50:16.638 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T10:50:16.683 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-30T10:50:16.708 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:19.0.0-1864.g5dc2ea30.el9.x86_64 5/6 2024-03-30T10:50:21.445 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1864.g5dc2ea30.el9.x86_64 5/6 2024-03-30T10:50:21.518 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:19.0.0-1864.g5dc2ea30.el9.x86_64 6/6 2024-03-30T10:50:23.896 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:19.0.0-1864.g5dc2ea30.el9.x86_64 6/6 2024-03-30T10:50:23.896 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/6 2024-03-30T10:50:23.896 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2/6 2024-03-30T10:50:23.896 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-30T10:50:23.897 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-30T10:50:23.897 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-16.el9.x86_64 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-30T10:50:24.235 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-30T10:50:24.236 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-30T10:50:24.236 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:24.236 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:24.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2024-03-30T10:50:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:42 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:25.313 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:25.315 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-30T10:50:25.315 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:50:25.315 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 6.6 k 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 2.1 M 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 1.5 M 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 248 k 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 4.7 M 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 17 M 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-30T10:50:25.316 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-30T10:50:25.317 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:25.318 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:50:25.319 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-30T10:50:25.319 INFO:teuthology.orchestra.run.smithi003.stdout:Install 36 Packages 2024-03-30T10:50:25.319 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:25.319 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 30 M 2024-03-30T10:50:25.320 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 107 M 2024-03-30T10:50:25.320 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:50:26.191 INFO:teuthology.orchestra.run.smithi003.stdout:(1/36): ceph-19.0.0-1864.g5dc2ea30.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-30T10:50:26.341 INFO:teuthology.orchestra.run.smithi003.stdout:(2/36): ceph-mds-19.0.0-1864.g5dc2ea30.el9.x86_ 5.5 MB/s | 2.1 MB 00:00 2024-03-30T10:50:26.383 INFO:teuthology.orchestra.run.smithi003.stdout:(3/36): ceph-mgr-19.0.0-1864.g5dc2ea30.el9.x86_ 3.4 MB/s | 1.5 MB 00:00 2024-03-30T10:50:26.466 INFO:teuthology.orchestra.run.smithi003.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1864.g5dc2 2.9 MB/s | 248 kB 00:00 2024-03-30T10:50:26.759 INFO:teuthology.orchestra.run.smithi003.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 640 kB/s | 187 kB 00:00 2024-03-30T10:50:26.909 INFO:teuthology.orchestra.run.smithi003.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-03-30T10:50:26.959 INFO:teuthology.orchestra.run.smithi003.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-30T10:50:27.118 INFO:teuthology.orchestra.run.smithi003.stdout:(8/36): ceph-osd-19.0.0-1864.g5dc2ea30.el9.x86_ 22 MB/s | 17 MB 00:00 2024-03-30T10:50:27.143 INFO:teuthology.orchestra.run.smithi003.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 258 kB/s | 47 kB 00:00 2024-03-30T10:50:27.168 INFO:teuthology.orchestra.run.smithi003.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-30T10:50:27.210 INFO:teuthology.orchestra.run.smithi003.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-30T10:50:27.277 INFO:teuthology.orchestra.run.smithi003.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-30T10:50:27.302 INFO:teuthology.orchestra.run.smithi003.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-30T10:50:27.327 INFO:teuthology.orchestra.run.smithi003.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-30T10:50:27.353 INFO:teuthology.orchestra.run.smithi003.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-30T10:50:27.420 INFO:teuthology.orchestra.run.smithi003.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 335 kB/s | 22 kB 00:00 2024-03-30T10:50:27.512 INFO:teuthology.orchestra.run.smithi003.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 680 kB/s | 125 kB 00:00 2024-03-30T10:50:27.762 INFO:teuthology.orchestra.run.smithi003.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 174 kB/s | 43 kB 00:00 2024-03-30T10:50:27.870 INFO:teuthology.orchestra.run.smithi003.stdout:(19/36): ceph-mon-19.0.0-1864.g5dc2ea30.el9.x86 2.8 MB/s | 4.7 MB 00:01 2024-03-30T10:50:27.896 INFO:teuthology.orchestra.run.smithi003.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 318 kB/s | 151 kB 00:00 2024-03-30T10:50:27.946 INFO:teuthology.orchestra.run.smithi003.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 934 kB/s | 172 kB 00:00 2024-03-30T10:50:28.005 INFO:teuthology.orchestra.run.smithi003.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-03-30T10:50:28.030 INFO:teuthology.orchestra.run.smithi003.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-30T10:50:28.055 INFO:teuthology.orchestra.run.smithi003.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-30T10:50:28.080 INFO:teuthology.orchestra.run.smithi003.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-30T10:50:28.122 INFO:teuthology.orchestra.run.smithi003.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 293 kB/s | 19 kB 00:00 2024-03-30T10:50:28.147 INFO:teuthology.orchestra.run.smithi003.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-30T10:50:28.174 INFO:teuthology.orchestra.run.smithi003.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-30T10:50:28.198 INFO:teuthology.orchestra.run.smithi003.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-30T10:50:28.232 INFO:teuthology.orchestra.run.smithi003.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-30T10:50:28.257 INFO:teuthology.orchestra.run.smithi003.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-30T10:50:28.282 INFO:teuthology.orchestra.run.smithi003.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-30T10:50:28.307 INFO:teuthology.orchestra.run.smithi003.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-03-30T10:50:28.333 INFO:teuthology.orchestra.run.smithi003.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-30T10:50:28.366 INFO:teuthology.orchestra.run.smithi003.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-30T10:50:28.391 INFO:teuthology.orchestra.run.smithi003.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-30T10:50:28.397 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:50:28.397 INFO:teuthology.orchestra.run.smithi003.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-03-30T10:50:28.421 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:50:28.468 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:50:28.468 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:50:28.892 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:50:28.892 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:50:30.237 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:50:30.311 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-30T10:50:30.351 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-30T10:50:30.380 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-30T10:50:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-30T10:50:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-30T10:50:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-30T10:50:30.623 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-30T10:50:30.701 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-30T10:50:30.748 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-30T10:50:30.821 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-30T10:50:30.874 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-30T10:50:30.912 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-30T10:50:30.979 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-30T10:50:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-30T10:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-30T10:50:31.282 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-30T10:50:31.321 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-30T10:50:31.355 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-30T10:50:31.385 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-30T10:50:31.463 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-30T10:50:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T10:50:31.528 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-30T10:50:31.569 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-30T10:50:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-30T10:50:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-30T10:50:31.731 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-30T10:50:31.778 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-30T10:50:31.853 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-30T10:50:31.995 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-30T10:50:32.126 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T10:50:32.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1864.g5dc2ea30.el9. 31/36 2024-03-30T10:50:32.315 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:19.0.0-1864.g5dc2ea30.el9.x86_64 32/36 2024-03-30T10:50:32.340 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1864.g5dc2ea30.el9.x86_64 32/36 2024-03-30T10:50:32.340 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:50:32.340 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-30T10:50:32.340 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T10:50:32.341 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-30T10:50:32.341 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:33.874 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 33/36 2024-03-30T10:50:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 33/36 2024-03-30T10:50:33.900 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:50:33.900 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-30T10:50:33.900 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T10:50:33.900 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-30T10:50:33.900 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:19.0.0-1864.g5dc2ea30.el9.x86_64 34/36 2024-03-30T10:50:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:19.0.0-1864.g5dc2ea30.el9.x86_64 34/36 2024-03-30T10:50:34.390 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:50:34.390 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-30T10:50:34.390 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T10:50:34.390 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-30T10:50:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:19.0.0-1864.g5dc2ea30.el9.x86_64 35/36 2024-03-30T10:50:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:19.0.0-1864.g5dc2ea30.el9.x86_64 35/36 2024-03-30T10:50:34.623 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:50:34.623 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-30T10:50:34.624 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T10:50:34.624 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-30T10:50:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:34.693 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:19.0.0-1864.g5dc2ea30.el9.x86_64 36/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:19.0.0-1864.g5dc2ea30.el9.x86_64 36/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:19.0.0-1864.g5dc2ea30.el9.x86_64 3/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:19.0.0-1864.g5dc2ea30.el9.x86_64 4/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 5/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1864.g5dc2ea30.el9. 6/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-30T10:50:36.509 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-30T10:50:36.510 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-30T10:50:36.511 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-30T10:50:36.511 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-30T10:50:36.512 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:37.620 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: lua-5.4.4-4.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-30T10:50:37.621 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-30T10:50:37.622 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-30T10:50:37.623 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-30T10:50:37.623 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-30T10:50:37.623 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-30T10:50:37.623 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:37.623 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:38.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2024-03-30T10:50:38.607 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:55 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:38.708 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:50:38.763 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:38.764 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:50:38.764 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:38.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2024-03-30T10:50:39.406 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:56 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:39.563 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 748 k 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 748 k 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 755 k 2024-03-30T10:50:39.564 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:50:40.389 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-19.0.0-1864.g5dc2ea30.el9.noarch.rpm 908 kB/s | 748 kB 00:00 2024-03-30T10:50:40.390 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:50:40.390 INFO:teuthology.orchestra.run.smithi003.stdout:Total 907 kB/s | 748 kB 00:00 2024-03-30T10:50:40.390 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:50:40.396 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:50:40.396 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:50:40.402 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:50:40.402 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:50:40.480 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:50:40.873 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/1 2024-03-30T10:50:40.974 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/1 2024-03-30T10:50:41.345 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/1 2024-03-30T10:50:41.647 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/1 2024-03-30T10:50:41.647 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:41.647 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:50:41.648 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:50:41.648 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:41.648 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:41.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2024-03-30T10:50:42.453 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:59 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:42.610 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:42.610 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:42.610 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-30T10:50:42.610 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:42.610 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:50:42.610 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 145 k 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 145 k 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 436 k 2024-03-30T10:50:42.611 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:50:42.960 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-19.0.0-1864.g5dc2ea 413 kB/s | 145 kB 00:00 2024-03-30T10:50:42.961 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:50:42.961 INFO:teuthology.orchestra.run.smithi003.stdout:Total 412 kB/s | 145 kB 00:00 2024-03-30T10:50:42.961 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:50:42.969 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:50:42.970 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:50:43.023 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:50:43.024 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:50:43.179 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:50:43.245 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1864.g5dc2ea30. 1/1 2024-03-30T10:50:43.269 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1864.g5dc2ea30. 1/1 2024-03-30T10:50:43.269 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:50:43.269 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-30T10:50:43.269 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:44.249 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1864.g5dc2ea30. 1/1 2024-03-30T10:50:44.249 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:44.249 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:50:44.249 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:50:44.249 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:44.249 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:44.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2024-03-30T10:50:45.110 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:45.212 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:50:45.266 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:45.270 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:50:45.270 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:45.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2024-03-30T10:50:45.911 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:02 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:46.066 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 1.8 M 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 31 k 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 16 k 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-30T10:50:46.068 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-30T10:50:46.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:46.070 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.9 M 2024-03-30T10:50:46.070 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-30T10:50:46.070 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:50:46.597 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1864.g5d 90 kB/s | 31 kB 00:00 2024-03-30T10:50:46.673 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1864.g5dc 39 kB/s | 16 kB 00:00 2024-03-30T10:50:46.748 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.2 MB/s | 238 kB 00:00 2024-03-30T10:50:46.831 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-30T10:50:47.132 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.2 MB/s | 863 kB 00:00 2024-03-30T10:50:47.232 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-30T10:50:47.257 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-30T10:50:47.416 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-30T10:50:47.441 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 786 kB/s | 144 kB 00:00 2024-03-30T10:50:47.491 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 409 kB/s | 31 kB 00:00 2024-03-30T10:50:47.533 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-30T10:50:47.726 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1864.g5dc2ea 1.2 MB/s | 1.8 MB 00:01 2024-03-30T10:50:47.728 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:50:47.728 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.1 MB/s | 6.9 MB 00:01 2024-03-30T10:50:47.747 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:50:47.764 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:50:47.764 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:50:47.944 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:50:47.945 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:50:48.595 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:50:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-30T10:50:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-30T10:50:48.751 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-30T10:50:48.779 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-30T10:50:48.890 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-30T10:50:49.018 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-30T10:50:49.110 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-30T10:50:49.159 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-30T10:50:49.188 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-30T10:50:49.219 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1864.g5dc2ea30.el9 10/12 2024-03-30T10:50:49.473 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1864.g5dc2ea30.el 11/12 2024-03-30T10:50:49.518 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1864.g5dc2ea30.el9.noa 12/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1864.g5dc2ea30.el9.noa 12/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1864.g5dc2ea30.el 1/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1864.g5dc2ea30.el9.noa 2/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1864.g5dc2ea30.el9 3/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-30T10:50:50.394 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-30T10:50:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-30T10:50:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-30T10:50:50.395 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-30T10:50:50.746 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-30T10:50:50.746 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:50.746 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:50.747 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:50:50.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-30T10:50:51.554 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:08 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:50:51.711 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:50:51.712 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-30T10:50:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 7.4 M 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-30T10:50:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-30T10:50:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:50:51.715 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2024-03-30T10:50:51.715 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2024-03-30T10:50:51.715 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:50:52.114 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 211 kB/s | 32 kB 00:00 2024-03-30T10:50:52.147 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-03-30T10:50:52.189 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 855 kB/s | 34 kB 00:00 2024-03-30T10:50:52.240 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-30T10:50:52.423 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-30T10:50:52.490 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-30T10:50:52.515 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-30T10:50:52.874 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-30T10:50:52.999 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-30T10:50:53.091 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 902 kB/s | 195 kB 00:00 2024-03-30T10:50:53.241 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 5.8 MB/s | 7.4 MB 00:01 2024-03-30T10:50:53.275 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 891 kB/s | 245 kB 00:00 2024-03-30T10:50:53.279 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:50:53.279 INFO:teuthology.orchestra.run.smithi003.stdout:Total 24 MB/s | 37 MB 00:01 2024-03-30T10:50:53.320 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:50:53.345 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:50:53.345 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:50:53.716 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:50:53.716 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:50:54.441 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:50:54.501 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-30T10:50:54.538 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-30T10:50:54.636 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-30T10:50:54.683 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-30T10:50:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T10:50:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T10:50:54.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-30T10:50:55.674 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-30T10:50:55.780 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-30T10:50:57.077 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-30T10:50:58.150 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-30T10:50:58.180 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1864.g5dc2e 12/12 2024-03-30T10:51:00.850 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1864.g5dc2e 12/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1864.g5dc2e 1/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-30T10:51:00.851 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-30T10:51:01.312 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-30T10:51:01.312 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:01.312 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:01.312 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:51:01.312 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-30T10:51:01.312 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:01.313 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:01.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2024-03-30T10:51:02.241 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:19 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:02.398 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:02.399 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-30T10:51:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:51:02.399 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-30T10:51:02.399 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 49 k 2024-03-30T10:51:02.399 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:02.400 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-30T10:51:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2024-03-30T10:51:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2024-03-30T10:51:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-30T10:51:02.401 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:02.588 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2024-03-30T10:51:02.613 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-03-30T10:51:02.663 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-30T10:51:02.689 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-30T10:51:02.706 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): ceph-mgr-rook-19.0.0-1864.g5dc2ea30.el9 195 kB/s | 49 kB 00:00 2024-03-30T10:51:02.731 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-30T10:51:02.756 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 642 kB/s | 43 kB 00:00 2024-03-30T10:51:02.865 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-03-30T10:51:02.882 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-03-30T10:51:03.023 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 418 kB/s | 59 kB 00:00 2024-03-30T10:51:03.049 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 693 kB/s | 203 kB 00:00 2024-03-30T10:51:03.165 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 632 kB/s | 90 kB 00:00 2024-03-30T10:51:03.316 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-03-30T10:51:03.318 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:03.318 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-30T10:51:03.319 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:03.335 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:03.336 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:03.443 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:03.443 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:03.967 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:04.148 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-30T10:51:04.210 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-30T10:51:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-30T10:51:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-30T10:51:04.360 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-30T10:51:04.458 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-30T10:51:04.574 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-30T10:51:04.629 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-30T10:51:05.390 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-30T10:51:05.443 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-30T10:51:05.477 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-30T10:51:05.525 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-30T10:51:05.555 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:19.0.0-1864.g5dc2ea30.el9.noarch 13/13 2024-03-30T10:51:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1864.g5dc2ea30.el9.noarch 13/13 2024-03-30T10:51:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/13 2024-03-30T10:51:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-30T10:51:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-30T10:51:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-30T10:51:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-30T10:51:06.786 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-30T10:51:06.787 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:06.788 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:07.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2024-03-30T10:51:07.617 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:24 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:07.777 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:07.778 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-30T10:51:07.778 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:51:07.778 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-30T10:51:07.778 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:07.778 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 138 k 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-30T10:51:07.779 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:07.780 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.8 M 2024-03-30T10:51:07.780 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2024-03-30T10:51:07.780 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:08.461 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-30T10:51:08.527 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 764 kB/s | 248 kB 00:00 2024-03-30T10:51:08.590 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1864.g5dc2ea30.e 361 kB/s | 138 kB 00:00 2024-03-30T10:51:08.861 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-03-30T10:51:08.886 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 170 kB/s | 51 kB 00:00 2024-03-30T10:51:08.978 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-30T10:51:08.981 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:08.981 INFO:teuthology.orchestra.run.smithi003.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-30T10:51:08.995 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:09.006 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:09.006 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:09.099 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:09.099 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:09.405 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:09.575 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-30T10:51:09.631 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-30T10:51:10.195 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-30T10:51:10.277 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-30T10:51:10.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-30T10:51:10.375 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 6/6 2024-03-30T10:51:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 6/6 2024-03-30T10:51:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/6 2024-03-30T10:51:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-30T10:51:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-30T10:51:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-30T10:51:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-30T10:51:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:11.494 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:11.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2024-03-30T10:51:12.315 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:29 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:12.473 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 888 k 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:12.474 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:12.475 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:12.475 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-30T10:51:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:12.475 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 966 k 2024-03-30T10:51:12.475 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.8 M 2024-03-30T10:51:12.475 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:12.596 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 654 kB/s | 79 kB 00:00 2024-03-30T10:51:13.380 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-19.0.0-1864.g5dc2ea30.el9.x86_ 982 kB/s | 888 kB 00:00 2024-03-30T10:51:13.381 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:13.381 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.0 MB/s | 966 kB 00:00 2024-03-30T10:51:13.381 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:13.407 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:13.407 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:13.488 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:13.488 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:13.676 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:13.827 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-30T10:51:13.907 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2/2 2024-03-30T10:51:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2/2 2024-03-30T10:51:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/2 2024-03-30T10:51:14.882 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-30T10:51:14.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:14.882 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:14.882 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-30T10:51:14.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:14.882 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:15.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2024-03-30T10:51:15.741 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:15.898 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:19.0.0-1864.g5dc2ea30.el9 ceph-noarch 264 k 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:15.899 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-30T10:51:15.900 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:15.900 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 332 k 2024-03-30T10:51:15.900 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.5 M 2024-03-30T10:51:15.900 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:16.025 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 551 kB/s | 69 kB 00:00 2024-03-30T10:51:16.484 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-volume-19.0.0-1864.g5dc2ea30.el9.no 452 kB/s | 264 kB 00:00 2024-03-30T10:51:16.485 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:16.485 INFO:teuthology.orchestra.run.smithi003.stdout:Total 568 kB/s | 332 kB 00:00 2024-03-30T10:51:16.485 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:16.495 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:16.495 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:16.535 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:16.535 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:16.686 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:16.881 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-30T10:51:16.920 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:19.0.0-1864.g5dc2ea30.el9.noarch 2/2 2024-03-30T10:51:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:19.0.0-1864.g5dc2ea30.el9.noarch 2/2 2024-03-30T10:51:16.936 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:51:16.936 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-30T10:51:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:17.798 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:19.0.0-1864.g5dc2ea30.el9.noarch 1/2 2024-03-30T10:51:18.105 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-30T10:51:18.105 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:18.106 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:18.106 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:19.0.0-1864.g5dc2ea30.el9.noarch 2024-03-30T10:51:18.106 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-30T10:51:18.106 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:18.106 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:18.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2024-03-30T10:51:19.020 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:36 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:19.178 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 127 k 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 127 k 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 456 k 2024-03-30T10:51:19.179 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:19.514 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-19.0.0-1864.g5dc2ea30.el9.x86_64 379 kB/s | 127 kB 00:00 2024-03-30T10:51:19.515 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:19.515 INFO:teuthology.orchestra.run.smithi003.stdout:Total 377 kB/s | 127 kB 00:00 2024-03-30T10:51:19.515 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:19.521 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:19.521 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:19.580 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:19.580 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:19.743 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:19.857 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:20.314 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:20.610 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:20.611 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:20.611 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:20.611 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:51:20.611 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:20.611 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:20.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2024-03-30T10:51:21.437 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:21.539 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:21.595 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:21.596 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:21.596 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:21.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2024-03-30T10:51:22.239 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:22.397 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:22.398 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-30T10:51:22.399 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:22.399 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 31 k 2024-03-30T10:51:22.399 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:22.399 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:22.400 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:22.400 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:51:22.400 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:22.400 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 k 2024-03-30T10:51:22.400 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 140 k 2024-03-30T10:51:22.400 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:22.884 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-19.0.0-1864.g5dc2ea30.el9.x86_6 64 kB/s | 31 kB 00:00 2024-03-30T10:51:22.884 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:22.884 INFO:teuthology.orchestra.run.smithi003.stdout:Total 64 kB/s | 31 kB 00:00 2024-03-30T10:51:22.885 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:22.888 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:22.888 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:22.918 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:22.919 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:23.065 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:23.186 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:23.577 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:23.832 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:51:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:23.832 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:24.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2024-03-30T10:51:24.580 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:24.682 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:24.737 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:24.738 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:24.738 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:24.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2024-03-30T10:51:25.368 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:25.471 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:25.526 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:25.527 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:25.527 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:25.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2024-03-30T10:51:26.153 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:26.254 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:26.309 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:26.310 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:26.310 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:26.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2024-03-30T10:51:26.951 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:27.055 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:27.111 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:27.112 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:27.112 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:27.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2024-03-30T10:51:27.739 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:27.841 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:27.895 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:27.896 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:27.896 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:27.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2024-03-30T10:51:28.534 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:45 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:28.637 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 is already installed. 2024-03-30T10:51:28.694 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:28.695 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-30T10:51:28.695 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:28.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2024-03-30T10:51:29.334 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:46 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:29.492 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 85 k 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:29.493 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 85 k 2024-03-30T10:51:29.494 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 231 k 2024-03-30T10:51:29.494 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:30.104 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-19.0.0-1864.g5dc2ea30.el9.x86_64.rpm 140 kB/s | 85 kB 00:00 2024-03-30T10:51:30.104 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:30.105 INFO:teuthology.orchestra.run.smithi003.stdout:Total 140 kB/s | 85 kB 00:00 2024-03-30T10:51:30.105 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:30.113 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:30.113 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:30.171 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:30.172 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:30.310 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:30.883 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:31.154 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:51:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:31.154 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:31.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2024-03-30T10:51:31.978 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:32.137 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:32.137 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 3.1 M 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:51:32.138 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:32.142 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.1 M 2024-03-30T10:51:32.142 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 13 M 2024-03-30T10:51:32.142 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:35.048 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-19.0.0-1864.g5dc2ea30.el9.x86_64.rpm 1.1 MB/s | 3.1 MB 00:02 2024-03-30T10:51:35.048 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:35.049 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.1 MB/s | 3.1 MB 00:02 2024-03-30T10:51:35.049 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:35.061 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:35.061 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:35.141 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:35.141 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:35.589 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:35.665 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:35.690 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-30T10:51:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-30T10:51:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T10:51:35.690 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-30T10:51:35.691 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:36.795 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:36.795 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:36.795 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:36.795 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:51:36.795 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:36.796 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:37.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2024-03-30T10:51:37.615 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Mar 2024 10:49:43 AM UTC. 2024-03-30T10:51:37.773 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-30T10:51:37.773 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:37.773 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:19.0.0-1864.g5dc2ea30.el9 ceph 170 k 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 170 k 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 490 k 2024-03-30T10:51:37.774 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-30T10:51:38.501 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-19.0.0-1864.g5dc2ea30.el9.x86_64.rpm 235 kB/s | 170 kB 00:00 2024-03-30T10:51:38.501 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-30T10:51:38.502 INFO:teuthology.orchestra.run.smithi003.stdout:Total 234 kB/s | 170 kB 00:00 2024-03-30T10:51:38.502 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-30T10:51:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-30T10:51:38.511 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-30T10:51:38.573 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-30T10:51:38.573 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-30T10:51:38.730 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-30T10:51:38.852 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:39.260 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:39.574 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 1/1 2024-03-30T10:51:39.575 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:39.575 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-30T10:51:39.575 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:19.0.0-1864.g5dc2ea30.el9.x86_64 2024-03-30T10:51:39.575 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-30T10:51:39.575 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-30T10:51:39.803 DEBUG:teuthology.parallel:result is None 2024-03-30T10:51:39.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T10:51:40.023 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T10:51:40.045 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-1864.g5dc2ea30.el9 2024-03-30T10:51:40.045 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864.g5dc2ea30.el9 2024-03-30T10:51:40.045 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864.g5dc2ea30 is installed. 2024-03-30T10:51:40.047 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T10:51:40.048 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:51:40.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T10:51:40.113 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T10:51:40.113 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:51:40.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T10:51:40.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T10:51:40.247 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T10:51:40.247 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:51:40.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T10:51:40.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T10:51:40.381 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T10:51:40.381 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:51:40.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T10:51:40.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T10:51:40.514 INFO:teuthology.run_tasks:Running task workunit... 2024-03-30T10:51:40.529 INFO:tasks.workunit:Pulling workunits from ref 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T10:51:40.529 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T10:51:40.530 INFO:tasks.workunit:timeout=3h 2024-03-30T10:51:40.530 INFO:tasks.workunit:cleanup=True 2024-03-30T10:51:40.530 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T10:51:40.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T10:51:40.568 INFO:teuthology.orchestra.run.smithi003.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-30T10:51:40.568 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-30T10:51:40.622 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-30T10:51:40.623 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-30T10:51:40.677 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T10:51:40.732 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T10:52:23.753 INFO:tasks.workunit.client.0.smithi003.stderr:Updating files: 99% (12040/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-03-30T10:52:24.657 INFO:tasks.workunit.client.0.smithi003.stderr:Note: switching to '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'. 2024-03-30T10:52:24.657 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: git switch -c 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:Or undo this operation with: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: git switch - 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T10:52:24.658 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-03-30T10:52:24.659 INFO:tasks.workunit.client.0.smithi003.stderr:HEAD is now at 5dc2ea3016d test/ceph_test_rados: fix seeking over virtual, in-memory objects 2024-03-30T10:52:24.663 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T10:52:24.719 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-30T10:52:24.719 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T10:52:24.774 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-03-30T10:52:24.775 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-03-30T10:52:24.775 DEBUG:teuthology.orchestra.run.smithi003:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-03-30T10:52:24.845 INFO:tasks.workunit.client.0.smithi003.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-30T10:52:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-03-30T10:52:24.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-03-30T10:52:24.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none ' 2024-03-30T10:52:24.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-03-30T10:52:24.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-30T10:52:24.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-03-30T10:52:24.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:52:24.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:52:24.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:52:24.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T10:52:24.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T10:52:24.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T10:52:24.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T10:52:24.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T10:52:24.860 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T10:52:24.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:52:24.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T10:52:24.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T10:52:24.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:52:24.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T10:52:24.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T10:52:24.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-30T10:52:24.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T10:52:24.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:52:24.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:52:24.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28888 2024-03-30T10:52:24.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T10:52:24.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T10:52:24.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-03-30T10:52:24.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T10:52:24.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:52:24.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:52:24.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.28888 2024-03-30T10:52:24.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-03-30T10:52:24.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-03-30T10:52:24.869 INFO:tasks.workunit.client.0.smithi003.stdout:Dir: td/osd-mapper 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-03-30T10:52:24.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-30T10:52:24.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-30T10:52:24.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-30T10:52:26.554 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 034bbaab-269a-4300-a6fc-40bf980cc437 2024-03-30T10:52:26.814 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:52:26.907 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:26.907+0000 7f0145c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:52:26.909 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:26.909+0000 7f0145c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:52:26.910 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:26.910+0000 7f0145c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:52:26.910 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:26.911+0000 7f0145c13640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-30T10:52:26.910 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:26.911+0000 7f0145c13640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-30T10:52:26.910 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:26.911+0000 7f0145c13640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-30T10:52:27.007 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:27.008+0000 7f0145c13640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-03-30T10:52:32.972 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T10:52:33.342 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T10:52:33.370 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:33.371+0000 7fb7ac59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:52:33.372 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:33.373+0000 7fb7ac59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:52:33.373 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:33.374+0000 7fb7ac59e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:52:33.624 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:52:33.891 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:33.892+0000 7fb7ac59e640 -1 Falling back to public interface 2024-03-30T10:52:34.903 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:52:36.193 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:52:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:52:36.227+0000 7fb7ac59e640 -1 osd.0 0 log_to_monitors true 2024-03-30T10:52:37.634 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:52:38.930 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T10:52:40.221 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T10:52:41.511 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T10:52:42.897 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T10:52:44.189 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T10:52:45.479 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T10:52:46.770 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T10:52:48.458 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T10:52:49.914 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T10:52:51.379 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T10:52:53.050 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T10:52:54.577 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T10:52:56.296 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T10:52:58.048 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T10:53:00.096 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T10:53:00.407 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2252884403,v1:127.0.0.1:6803/2252884403] [v2:127.0.0.1:6804/2252884403,v1:127.0.0.1:6805/2252884403] exists,up 034bbaab-269a-4300-a6fc-40bf980cc437 2024-03-30T10:53:00.411 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 40300248-f7e0-48fe-bd94-3f130e2aad2c 2024-03-30T10:53:00.790 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:53:00.831 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.832+0000 7f8c39415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:00.874 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.875+0000 7f8c39415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:00.875 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.876+0000 7f8c39415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:00.876 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.876+0000 7f8c39415640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-30T10:53:00.876 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.876+0000 7f8c39415640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-30T10:53:00.876 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.876+0000 7f8c39415640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-30T10:53:00.949 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:00.950+0000 7f8c39415640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-03-30T10:53:06.940 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T10:53:07.304 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T10:53:07.333 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:07.333+0000 7f6efded9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:07.334 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:07.335+0000 7f6efded9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:07.335 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:07.336+0000 7f6efded9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:07.584 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:53:07.878 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:07.878+0000 7f6efded9640 -1 Falling back to public interface 2024-03-30T10:53:08.867 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:53:10.155 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:53:10.256 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:10.256+0000 7f6efded9640 -1 osd.1 0 log_to_monitors true 2024-03-30T10:53:11.474 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:53:12.770 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T10:53:14.185 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T10:53:15.478 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T10:53:16.773 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T10:53:18.135 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T10:53:19.451 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T10:53:20.740 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T10:53:22.193 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T10:53:24.145 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T10:53:25.572 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T10:53:27.233 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T10:53:28.720 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T10:53:30.070 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T10:53:31.372 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T10:53:33.143 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T10:53:33.434 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/110649129,v1:127.0.0.1:6811/110649129] [v2:127.0.0.1:6812/110649129,v1:127.0.0.1:6813/110649129] exists,up 40300248-f7e0-48fe-bd94-3f130e2aad2c 2024-03-30T10:53:33.438 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 94f587db-db62-4cab-aa70-34d58911e634 2024-03-30T10:53:33.786 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:53:33.827 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.828+0000 7f0974b93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:33.862 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.862+0000 7f0974b93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:33.863 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.863+0000 7f0974b93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:33.863 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.864+0000 7f0974b93640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-30T10:53:33.863 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.864+0000 7f0974b93640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-30T10:53:33.863 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.864+0000 7f0974b93640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-30T10:53:33.928 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:33.929+0000 7f0974b93640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-03-30T10:53:39.961 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T10:53:40.324 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T10:53:40.352 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:40.352+0000 7f3554ea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:40.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:40.359+0000 7f3554ea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:40.360 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:40.360+0000 7f3554ea2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:53:40.639 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:53:40.892 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:40.893+0000 7f3554ea2640 -1 Falling back to public interface 2024-03-30T10:53:41.922 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:53:43.223 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:53:43.228 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:53:43.228+0000 7f3554ea2640 -1 osd.2 0 log_to_monitors true 2024-03-30T10:53:44.536 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:53:45.876 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T10:53:47.183 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T10:53:48.533 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T10:53:49.860 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T10:53:51.256 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T10:53:52.548 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T10:53:53.863 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T10:53:55.234 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T10:53:56.540 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T10:53:58.129 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T10:54:00.023 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T10:54:02.037 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T10:54:03.849 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T10:54:05.425 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T10:54:05.728 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/433433897,v1:127.0.0.1:6819/433433897] [v2:127.0.0.1:6820/433433897,v1:127.0.0.1:6821/433433897] exists,up 94f587db-db62-4cab-aa70-34d58911e634 2024-03-30T10:54:06.117 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'test' created 2024-03-30T10:54:07.935 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836498 2024-03-30T10:54:09.506 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672970 2024-03-30T10:54:09.789 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509443 2024-03-30T10:54:12.333 INFO:tasks.workunit.client.0.smithi003.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-03-30T10:54:12.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-03-30T10:54:12.448 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0: { 2024-03-30T10:54:12.448 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-30T10:54:12.448 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:12.463 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1: { 2024-03-30T10:54:12.463 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-30T10:54:12.463 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:12.476 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2: { 2024-03-30T10:54:12.476 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-30T10:54:12.476 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:12.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-03-30T10:54:12.599 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0: { 2024-03-30T10:54:12.599 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-30T10:54:12.599 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:12.613 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1: { 2024-03-30T10:54:12.613 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-30T10:54:12.613 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:12.627 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2: { 2024-03-30T10:54:12.627 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-30T10:54:12.627 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:12.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-03-30T10:54:12.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-03-30T10:54:12.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-03-30T10:54:12.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-03-30T10:54:12.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-03-30T10:54:12.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-30T10:54:12.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-30T10:54:12.950 INFO:tasks.workunit.client.0.smithi003.stdout:created pool test snap snap01 2024-03-30T10:54:13.192 INFO:tasks.workunit.client.0.smithi003.stdout:created pool test snap snap02 2024-03-30T10:54:13.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-03-30T10:54:13.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-30T10:54:13.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-30T10:54:13.654 INFO:tasks.workunit.client.0.smithi003.stdout:created pool test snap snap13 2024-03-30T10:54:13.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-03-30T10:54:13.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-30T10:54:13.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-30T10:54:14.073 INFO:tasks.workunit.client.0.smithi003.stdout:created pool test snap snap24 2024-03-30T10:54:14.391 INFO:tasks.workunit.client.0.smithi003.stdout:created pool test snap snap25 2024-03-30T10:54:14.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 22309 2024-03-30T10:54:14.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-03-30T10:54:14.687 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-30T10:54:14.687 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-03-30T10:54:14.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-03-30T10:54:14.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-30T10:54:14.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-30T10:54:15.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-03-30T10:54:15.274 INFO:tasks.workunit.client.0.smithi003.stdout:pgid is 1.3 (primary: osd.1) 2024-03-30T10:54:15.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-03-30T10:54:15.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-03-30T10:54:15.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-03-30T10:54:15.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-03-30T10:54:15.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-03-30T10:54:15.537 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs_brief 2024-03-30T10:54:15.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-03-30T10:54:15.548 INFO:tasks.workunit.client.0.smithi003.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-03-30T10:54:15.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.28888/ceph-osd.1.asok 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-03-30T10:54:15.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.28888/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-03-30T10:54:15.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:15.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout:{"success":true}{ 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "objxxx", 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: "clones": [ 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 2, 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [ 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 1, 2024-03-30T10:54:15.684 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap01" 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 2, 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap02" 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6, 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "overlaps": [] 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 3, 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [ 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.685 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 3, 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap13" 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6, 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "overlaps": [] 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 5, 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [ 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 4, 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap24" 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:15.686 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 5, 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap25" 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6, 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "overlaps": [] 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "id": "head", 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [], 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout: ] 2024-03-30T10:54:15.687 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:15.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-30T10:54:15.780 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:54:15.780 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": true, 2024-03-30T10:54:15.780 INFO:tasks.workunit.client.0.smithi003.stdout: "must": true, 2024-03-30T10:54:15.780 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "0.000000" 2024-03-30T10:54:15.780 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:15.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-03-30T10:54:18.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-03-30T10:54:16.211233+0000 26'7 26:47 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-30T10:54:16.211154+0000 26'7 2024-03-30T10:54:16.211154+0000 0 1 periodic scrub scheduled @ 2024-03-31T10:54:16.211154+0000 4 0 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:07.412086+0000 0'0 26:32 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:07.412285+0000 0'0 26:32 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:07.317580+0000 0'0 26:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:19.063 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:54:19.064 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:54:19.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-03-30T10:54:19.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-30T10:54:19.341 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:54:19.341 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-03-30T10:54:16.211233+0000 26'7 26:47 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-30T10:54:16.211154+0000 26'7 2024-03-30T10:54:16.211154+0000 0 1 periodic scrub scheduled @ 2024-03-31T10:54:16.211154+0000 4 0 2024-03-30T10:54:19.341 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:07.412086+0000 0'0 26:32 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:19.341 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:07.412285+0000 0'0 26:32 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:19.341 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:07.317580+0000 0'0 26:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:19.342 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:19.342 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:54:19.342 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:54:19.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-03-30T10:54:19.738 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T10:54:19.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-03-30T10:54:20.265 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T10:54:20.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-03-30T10:54:25.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-03-30T10:54:25.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-30T10:54:25.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:54:25.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:54:25.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:54:25.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:54:25.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:54:25.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:54:25.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:25.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bdev(0x560f0bc3c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 0 bdev(0x560f0bc3c000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bdev(0x560f0bc3c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bdev(0x560f0bc3c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 0 bdev(0x560f0bc3c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bdev(0x560f0bc3c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-30T10:54:25.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.613+0000 7f3405639980 1 bluefs mount 2024-03-30T10:54:25.626 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.626+0000 7f3405639980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.659+0000 7f3405639980 1 bluefs mount shared_bdev_used = 0 2024-03-30T10:54:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.659+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:25.663 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:25.663 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:25.663 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:25.663 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:25.663 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:25.663 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:25.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:25.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:25.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.662+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:25.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.663+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:25.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.663+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:25.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.663+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 freelist init 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 freelist _read_cfg 2024-03-30T10:54:25.698 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.697+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-30T10:54:25.699 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.698+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.39235e-07 2024-03-30T10:54:25.699 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.698+0000 7f3405639980 1 bluefs umount 2024-03-30T10:54:25.699 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.698+0000 7f3405639980 1 bdev(0x560f0bc3c700 td/osd-mapper/0/block) close 2024-03-30T10:54:25.882 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.882+0000 7f3405639980 1 bdev(0x560f0bc3c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-30T10:54:25.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.882+0000 7f3405639980 0 bdev(0x560f0bc3c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-30T10:54:25.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.882+0000 7f3405639980 1 bdev(0x560f0bc3c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:25.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.882+0000 7f3405639980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-30T10:54:25.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.882+0000 7f3405639980 1 bluefs mount 2024-03-30T10:54:25.893 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.892+0000 7f3405639980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:25.901 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.900+0000 7f3405639980 1 bluefs mount shared_bdev_used = 27459584 2024-03-30T10:54:25.901 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.900+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.902+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.902+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:25.903+0000 7f3405639980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:26.900 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:26.900+0000 7f3405639980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:26.902 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-30T10:54:26.902 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 0b 05 58 02 00 00 3f 31 a9 45 1d 20 42 c1 95 89 |..X...?1.E. B...| 2024-03-30T10:54:26.902 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 21 1f ba 07 23 62 00 00 00 00 1c 00 00 00 00 00 |!...#b..........| 2024-03-30T10:54:26.902 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-30T10:54:26.902 INFO:tasks.workunit.client.0.smithi003.stdout:* 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-30T10:54:26.903 INFO:tasks.workunit.client.0.smithi003.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-30T10:54:26.904 INFO:tasks.workunit.client.0.smithi003.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 03 4b ba ab 26 9a |re.........K..&.| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000230 43 00 a6 fc 40 bf 98 0c c4 37 00 00 00 00 00 00 |C...@....7......| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000240 00 00 1c 00 00 00 0a ef 07 66 16 2e 35 18 01 00 |.........f..5...| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:0000025e 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-30T10:54:26.905 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:0000003f 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:26.906 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:26.907 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:26.908 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:26.901+0000 7f3405639980 1 bluefs umount 2024-03-30T10:54:26.909 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:26.901+0000 7f3405639980 1 bdev(0x560f0bc3c700 td/osd-mapper/0/block) close 2024-03-30T10:54:27.137 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.137+0000 7f3405639980 1 freelist shutdown 2024-03-30T10:54:27.137 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.137+0000 7f3405639980 1 bdev(0x560f0bc3c000 td/osd-mapper/0/block) close 2024-03-30T10:54:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:27.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-30T10:54:27.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-03-30T10:54:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.404+0000 7fd1ea0d2980 1 bdev(0x55f2f149a000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-30T10:54:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.404+0000 7fd1ea0d2980 0 bdev(0x55f2f149a000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-30T10:54:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.404+0000 7fd1ea0d2980 1 bdev(0x55f2f149a000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.404+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-30T10:54:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.404+0000 7fd1ea0d2980 1 bdev(0x55f2f149a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-30T10:54:27.406 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.405+0000 7fd1ea0d2980 0 bdev(0x55f2f149a700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-30T10:54:27.406 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.405+0000 7fd1ea0d2980 1 bdev(0x55f2f149a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:27.406 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.405+0000 7fd1ea0d2980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-30T10:54:27.406 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.405+0000 7fd1ea0d2980 1 bluefs mount 2024-03-30T10:54:27.417 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.416+0000 7fd1ea0d2980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:27.425 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.424+0000 7fd1ea0d2980 1 bluefs mount shared_bdev_used = 0 2024-03-30T10:54:27.425 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.424+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:27.429 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:27.429 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.427+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 freelist init 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 freelist _read_cfg 2024-03-30T10:54:27.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.460+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-30T10:54:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.461+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.34059e-07 2024-03-30T10:54:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.461+0000 7fd1ea0d2980 1 bluefs umount 2024-03-30T10:54:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.461+0000 7fd1ea0d2980 1 bdev(0x55f2f149a700 td/osd-mapper/2/block) close 2024-03-30T10:54:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.667+0000 7fd1ea0d2980 1 bdev(0x55f2f149a700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-30T10:54:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.667+0000 7fd1ea0d2980 0 bdev(0x55f2f149a700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-30T10:54:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.667+0000 7fd1ea0d2980 1 bdev(0x55f2f149a700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:27.668 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.667+0000 7fd1ea0d2980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-30T10:54:27.668 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.667+0000 7fd1ea0d2980 1 bluefs mount 2024-03-30T10:54:27.675 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.674+0000 7fd1ea0d2980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:27.683 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.683+0000 7fd1ea0d2980 1 bluefs mount shared_bdev_used = 27459584 2024-03-30T10:54:27.683 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.683+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:27.686 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:27.687 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:27.687 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:27.687 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:27.687 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:27.685+0000 7fd1ea0d2980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:28.587 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:28.587+0000 7fd1ea0d2980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:28.588 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:28.588+0000 7fd1ea0d2980 1 bluefs umount 2024-03-30T10:54:28.588 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:28.588+0000 7fd1ea0d2980 1 bdev(0x55f2f149a700 td/osd-mapper/2/block) close 2024-03-30T10:54:28.827 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:28.828+0000 7fd1ea0d2980 1 freelist shutdown 2024-03-30T10:54:28.827 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:28.828+0000 7fd1ea0d2980 1 bdev(0x55f2f149a000 td/osd-mapper/2/block) close 2024-03-30T10:54:28.922 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:28.922 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:28.922 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:28.922 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:28.922 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:28.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:28.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-03-30T10:54:28.923 INFO:tasks.workunit.client.0.smithi003.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-03-30T10:54:28.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:28.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 0b 05 58 02 00 00 3f 31 a9 45 1d 20 42 c1 95 89 |..X...?1.E. B...| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 21 1f ba 07 23 62 02 00 00 00 1c 00 00 00 00 00 |!...#b..........| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:* 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-30T10:54:29.482 INFO:tasks.workunit.client.0.smithi003.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-30T10:54:29.483 INFO:tasks.workunit.client.0.smithi003.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 94 f5 87 db db 62 |re.............b| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000230 4c ab aa 70 34 d5 89 11 e6 34 00 00 00 00 00 00 |L..p4....4......| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000240 00 00 1c 00 00 00 4d ef 07 66 d3 f0 5a 03 01 00 |......M..f..Z...| 2024-03-30T10:54:29.484 INFO:tasks.workunit.client.0.smithi003.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:0000025e 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:29.485 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:0000003f 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:29.486 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:29.487 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:29.488 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-30T10:54:29.489 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:29.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-30T10:54:29.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-30T10:54:29.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-30T10:54:29.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-03-30T10:54:31.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-03-30T10:54:31.505 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-03-30T10:54:31.506 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-03-30T10:54:31.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:31.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-03-30T10:54:31.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-03-30T10:54:31.508 INFO:tasks.workunit.client.0.smithi003.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-03-30T10:54:31.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-30T10:54:31.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-30T10:54:31.532 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.532+0000 7f1de4240980 1 bdev(0x55b84187c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-30T10:54:31.532 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.532+0000 7f1de4240980 0 bdev(0x55b84187c000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.532+0000 7f1de4240980 1 bdev(0x55b84187c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.533+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.533+0000 7f1de4240980 1 bdev(0x55b84187c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.533+0000 7f1de4240980 0 bdev(0x55b84187c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.533+0000 7f1de4240980 1 bdev(0x55b84187c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.533+0000 7f1de4240980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-30T10:54:31.533 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.533+0000 7f1de4240980 1 bluefs mount 2024-03-30T10:54:31.542 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.542+0000 7f1de4240980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:31.551 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.552+0000 7f1de4240980 1 bluefs mount shared_bdev_used = 0 2024-03-30T10:54:31.552 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.552+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.554+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:31.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.555+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 freelist init 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 freelist _read_cfg 2024-03-30T10:54:31.560 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.560+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-30T10:54:31.561 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.561+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.39235e-07 2024-03-30T10:54:31.561 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.561+0000 7f1de4240980 1 bluefs umount 2024-03-30T10:54:31.561 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.561+0000 7f1de4240980 1 bdev(0x55b84187c700 td/osd-mapper/0/block) close 2024-03-30T10:54:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.795+0000 7f1de4240980 1 bdev(0x55b84187c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-30T10:54:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.795+0000 7f1de4240980 0 bdev(0x55b84187c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-30T10:54:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.795+0000 7f1de4240980 1 bdev(0x55b84187c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.795+0000 7f1de4240980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-30T10:54:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.795+0000 7f1de4240980 1 bluefs mount 2024-03-30T10:54:31.808 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.809+0000 7f1de4240980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:31.818 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.818+0000 7f1de4240980 1 bluefs mount shared_bdev_used = 9568256 2024-03-30T10:54:31.819 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.818+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:31.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.820+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:31.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.820+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:31.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.820+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:31.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.820+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.820+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:31.821+0000 7f1de4240980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:32.077 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:32.078+0000 7f1de4240980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:32.079 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 0b 05 58 02 00 00 3f 31 a9 45 1d 20 42 c1 95 89 |..X...?1.E. B...| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 21 1f ba 07 23 62 00 00 00 00 1c 00 00 00 00 00 |!...#b..........| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:* 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-30T10:54:32.080 INFO:tasks.workunit.client.0.smithi003.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-30T10:54:32.081 INFO:tasks.workunit.client.0.smithi003.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 03 4b ba ab 26 9a |re.........K..&.| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:00000230 43 00 a6 fc 40 bf 98 0c c4 37 00 00 00 00 00 00 |C...@....7......| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:00000240 00 00 1c 00 00 00 0a ef 07 66 16 2e 35 18 01 00 |.........f..5...| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout:0000025e 2024-03-30T10:54:32.082 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:32.083 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:0000003f 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-30T10:54:32.084 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:32.085 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:32.086 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:32.087 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.087 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:32.088 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.089 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-30T10:54:32.089 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.089 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-30T10:54:32.089 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:32.089 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:32.079+0000 7f1de4240980 1 bluefs umount 2024-03-30T10:54:32.089 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:32.079+0000 7f1de4240980 1 bdev(0x55b84187c700 td/osd-mapper/0/block) close 2024-03-30T10:54:32.314 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:32.315+0000 7f1de4240980 1 freelist shutdown 2024-03-30T10:54:32.314 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:32.315+0000 7f1de4240980 1 bdev(0x55b84187c000 td/osd-mapper/0/block) close 2024-03-30T10:54:32.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-30T10:54:32.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-30T10:54:32.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-30T10:54:33.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:33.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:33.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-30T10:54:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-30T10:54:33.594 INFO:tasks.workunit.client.0.smithi003.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:33.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-30T10:54:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.1ryw1U0SeW_the_val 2024-03-30T10:54:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.1ryw1U0SeW_the_val' 2024-03-30T10:54:33.595 INFO:tasks.workunit.client.0.smithi003.stdout:Value dumped in: /tmp/tmp.1ryw1U0SeW_the_val 2024-03-30T10:54:33.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.1ryw1U0SeW_the_val 2024-03-30T10:54:34.122 INFO:tasks.workunit.client.0.smithi003.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-30T10:54:34.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:34.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.1ryw1U0SeW_the_val 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-30T10:54:34.400 INFO:tasks.workunit.client.0.smithi003.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-30T10:54:34.401 INFO:tasks.workunit.client.0.smithi003.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-30T10:54:34.401 INFO:tasks.workunit.client.0.smithi003.stdout:0000073 2024-03-30T10:54:34.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-30T10:54:34.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:35.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.1ryw1U0SeW_the_val 2024-03-30T10:54:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-30T10:54:36.488 INFO:tasks.workunit.client.0.smithi003.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-03-30T10:54:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-03-30T10:54:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-30T10:54:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bdev(0x55ee94ea0000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 0 bdev(0x55ee94ea0000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bdev(0x55ee94ea0000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bdev(0x55ee94ea0700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 0 bdev(0x55ee94ea0700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bdev(0x55ee94ea0700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:36.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-30T10:54:36.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.512+0000 7f53a7377980 1 bluefs mount 2024-03-30T10:54:36.517 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.517+0000 7f53a7377980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:36.526 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.527+0000 7f53a7377980 1 bluefs mount shared_bdev_used = 0 2024-03-30T10:54:36.527 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.527+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.529+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.529+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.529+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.529+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.529+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.530+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 freelist init 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 freelist _read_cfg 2024-03-30T10:54:36.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.535+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-30T10:54:36.536 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.536+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 16 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.72206e-07 2024-03-30T10:54:36.536 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.536+0000 7f53a7377980 1 bluefs umount 2024-03-30T10:54:36.536 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.536+0000 7f53a7377980 1 bdev(0x55ee94ea0700 td/osd-mapper/1/block) close 2024-03-30T10:54:36.781 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.782+0000 7f53a7377980 1 bdev(0x55ee94ea0700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-30T10:54:36.781 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.782+0000 7f53a7377980 0 bdev(0x55ee94ea0700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-30T10:54:36.782 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.782+0000 7f53a7377980 1 bdev(0x55ee94ea0700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:36.782 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.782+0000 7f53a7377980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-30T10:54:36.782 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.782+0000 7f53a7377980 1 bluefs mount 2024-03-30T10:54:36.792 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.792+0000 7f53a7377980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:36.800 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.801+0000 7f53a7377980 1 bluefs mount shared_bdev_used = 9568256 2024-03-30T10:54:36.800 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.801+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:36.803 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.803+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:36.803 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.803+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:36.803 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.803+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:36.803 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.803+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.803+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:36.804 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:36.804+0000 7f53a7377980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:37.052 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:37.053+0000 7f53a7377980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:37.054 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-30T10:54:37.054 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 0b 05 58 02 00 00 3f 31 a9 45 1d 20 42 c1 95 89 |..X...?1.E. B...| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 21 1f ba 07 23 62 01 00 00 00 1c 00 00 00 00 00 |!...#b..........| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:* 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-30T10:54:37.055 INFO:tasks.workunit.client.0.smithi003.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-30T10:54:37.056 INFO:tasks.workunit.client.0.smithi003.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:00000220 72 65 1c 00 00 00 0a 00 00 00 40 30 02 48 f7 e0 |re........@0.H..| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:00000230 48 fe bd 94 3f 13 0e 2a ad 2c 00 00 00 00 00 00 |H...?..*.,......| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:00000240 00 00 1c 00 00 00 2b ef 07 66 a7 78 04 3a 01 00 |......+..f.x.:..| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:0000025e 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.057 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-30T10:54:37.058 INFO:tasks.workunit.client.0.smithi003.stdout:0000003f 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:37.059 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:37.060 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.061 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-30T10:54:37.062 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.062 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-30T10:54:37.062 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:37.062 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:37.054+0000 7f53a7377980 1 bluefs umount 2024-03-30T10:54:37.062 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:37.054+0000 7f53a7377980 1 bdev(0x55ee94ea0700 td/osd-mapper/1/block) close 2024-03-30T10:54:37.292 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:37.293+0000 7f53a7377980 1 freelist shutdown 2024-03-30T10:54:37.292 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:37.293+0000 7f53a7377980 1 bdev(0x55ee94ea0000 td/osd-mapper/1/block) close 2024-03-30T10:54:37.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-30T10:54:37.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-30T10:54:37.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-30T10:54:38.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:38.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:38.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-30T10:54:38.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-30T10:54:38.319 INFO:tasks.workunit.client.0.smithi003.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:38.319 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-30T10:54:38.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.2Mys283bxm_the_val 2024-03-30T10:54:38.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:38.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.2Mys283bxm_the_val' 2024-03-30T10:54:38.321 INFO:tasks.workunit.client.0.smithi003.stdout:Value dumped in: /tmp/tmp.2Mys283bxm_the_val 2024-03-30T10:54:38.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.2Mys283bxm_the_val 2024-03-30T10:54:38.922 INFO:tasks.workunit.client.0.smithi003.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-30T10:54:39.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:39.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.2Mys283bxm_the_val 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-30T10:54:39.368 INFO:tasks.workunit.client.0.smithi003.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-30T10:54:39.369 INFO:tasks.workunit.client.0.smithi003.stdout:0000073 2024-03-30T10:54:39.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-30T10:54:39.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:40.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.2Mys283bxm_the_val 2024-03-30T10:54:41.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:41.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:41.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-03-30T10:54:41.211 INFO:tasks.workunit.client.0.smithi003.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-03-30T10:54:41.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-03-30T10:54:41.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-30T10:54:41.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-30T10:54:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.237+0000 7f24c1039980 1 bdev(0x557418404000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-30T10:54:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.237+0000 7f24c1039980 0 bdev(0x557418404000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-30T10:54:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.237+0000 7f24c1039980 1 bdev(0x557418404000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:41.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.237+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-30T10:54:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.237+0000 7f24c1039980 1 bdev(0x557418404700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-30T10:54:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.237+0000 7f24c1039980 0 bdev(0x557418404700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-30T10:54:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.238+0000 7f24c1039980 1 bdev(0x557418404700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.238+0000 7f24c1039980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-30T10:54:41.238 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.238+0000 7f24c1039980 1 bluefs mount 2024-03-30T10:54:41.248 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.248+0000 7f24c1039980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:41.257 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.257+0000 7f24c1039980 1 bluefs mount shared_bdev_used = 0 2024-03-30T10:54:41.257 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.257+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:41.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:41.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:41.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:41.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:41.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:41.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.260+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 freelist init 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 freelist _read_cfg 2024-03-30T10:54:41.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.265+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-30T10:54:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.266+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 15 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 5.34059e-07 2024-03-30T10:54:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.266+0000 7f24c1039980 1 bluefs umount 2024-03-30T10:54:41.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.266+0000 7f24c1039980 1 bdev(0x557418404700 td/osd-mapper/2/block) close 2024-03-30T10:54:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.501+0000 7f24c1039980 1 bdev(0x557418404700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-30T10:54:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.501+0000 7f24c1039980 0 bdev(0x557418404700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-30T10:54:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.501+0000 7f24c1039980 1 bdev(0x557418404700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard supported 2024-03-30T10:54:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.501+0000 7f24c1039980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-30T10:54:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.501+0000 7f24c1039980 1 bluefs mount 2024-03-30T10:54:41.506 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.507+0000 7f24c1039980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-30T10:54:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.515+0000 7f24c1039980 1 bluefs mount shared_bdev_used = 9568256 2024-03-30T10:54:41.515 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.515+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.517+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-30T10:54:41.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-30T10:54:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-30T10:54:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-30T10:54:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-30T10:54:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-30T10:54:41.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.518+0000 7f24c1039980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-30T10:54:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.727+0000 7f24c1039980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-30T10:54:41.728 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-30T10:54:41.728 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 0b 05 58 02 00 00 3f 31 a9 45 1d 20 42 c1 95 89 |..X...?1.E. B...| 2024-03-30T10:54:41.728 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 21 1f ba 07 23 62 02 00 00 00 1c 00 00 00 00 00 |!...#b..........| 2024-03-30T10:54:41.728 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-30T10:54:41.728 INFO:tasks.workunit.client.0.smithi003.stdout:* 2024-03-30T10:54:41.728 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-30T10:54:41.729 INFO:tasks.workunit.client.0.smithi003.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-30T10:54:41.730 INFO:tasks.workunit.client.0.smithi003.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 94 f5 87 db db 62 |re.............b| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000230 4c ab aa 70 34 d5 89 11 e6 34 00 00 00 00 00 00 |L..p4....4......| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000240 00 00 1c 00 00 00 4d ef 07 66 d3 f0 5a 03 01 00 |......M..f..Z...| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:0000025e 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:41.731 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:0000003f 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-30T10:54:41.732 INFO:tasks.workunit.client.0.smithi003.stdout:00000047 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:41.733 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:41.734 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:0000003b 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:41.735 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.728+0000 7f24c1039980 1 bluefs umount 2024-03-30T10:54:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.728+0000 7f24c1039980 1 bdev(0x557418404700 td/osd-mapper/2/block) close 2024-03-30T10:54:41.963 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.964+0000 7f24c1039980 1 freelist shutdown 2024-03-30T10:54:41.963 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:41.964+0000 7f24c1039980 1 bdev(0x557418404000 td/osd-mapper/2/block) close 2024-03-30T10:54:42.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-30T10:54:42.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-30T10:54:42.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-30T10:54:42.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:42.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:42.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-30T10:54:42.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-30T10:54:42.794 INFO:tasks.workunit.client.0.smithi003.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:42.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-30T10:54:42.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.fdxpoIXFj3_the_val 2024-03-30T10:54:42.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:42.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.fdxpoIXFj3_the_val' 2024-03-30T10:54:42.796 INFO:tasks.workunit.client.0.smithi003.stdout:Value dumped in: /tmp/tmp.fdxpoIXFj3_the_val 2024-03-30T10:54:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.fdxpoIXFj3_the_val 2024-03-30T10:54:43.295 INFO:tasks.workunit.client.0.smithi003.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-30T10:54:43.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:43.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.fdxpoIXFj3_the_val 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-30T10:54:43.589 INFO:tasks.workunit.client.0.smithi003.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-30T10:54:43.590 INFO:tasks.workunit.client.0.smithi003.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-30T10:54:43.590 INFO:tasks.workunit.client.0.smithi003.stdout:0000073 2024-03-30T10:54:43.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-30T10:54:43.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:54:44.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.fdxpoIXFj3_the_val 2024-03-30T10:54:45.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:54:45.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-03-30T10:54:45.430 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-03-30T10:54:45.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:45.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:54:45.430 INFO:tasks.workunit.client.0.smithi003.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-03-30T10:54:45.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-03-30T10:54:45.431 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-03-30T10:54:45.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-03-30T10:54:45.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:45.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T10:54:45.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28888/$cluster-$name.asok' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28888/$cluster-$name.asok' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:54:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T10:54:45.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T10:54:45.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-03-30T10:54:45.438 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28888/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T10:54:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T10:54:45.477 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:45.477+0000 7f07b5f81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:45.478 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:45.479+0000 7f07b5f81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:45.480 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:45.480+0000 7f07b5f81640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:45.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T10:54:45.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:54:45.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T10:54:45.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:54:45.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:54:45.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:45.741 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:54:45.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:54:45.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:45.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:54:45.996 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:45.997+0000 7f07b5f81640 -1 Falling back to public interface 2024-03-30T10:54:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:54:47.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:54:47.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:47.072 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:54:47.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:54:47.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:47.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:54:47.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:54:48.197 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:48.198+0000 7f07b5f81640 -1 osd.0 28 log_to_monitors true 2024-03-30T10:54:48.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:54:48.375 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:54:48.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:48.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:54:48.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:48.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:54:48.743 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 31 up_thru 16 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/908779185,v1:127.0.0.1:6803/908779185] [v2:127.0.0.1:6804/908779185,v1:127.0.0.1:6805/908779185] exists,up 034bbaab-269a-4300-a6fc-40bf980cc437 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-03-30T10:54:48.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T10:54:48.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28888/$cluster-$name.asok' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28888/$cluster-$name.asok' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T10:54:48.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T10:54:48.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-03-30T10:54:48.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T10:54:48.750 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T10:54:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28888/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T10:54:48.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-03-30T10:54:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T10:54:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T10:54:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T10:54:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T10:54:48.782 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:48.783+0000 7f145f214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:48.815 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:48.815+0000 7f145f214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:48.816 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:48.817+0000 7f145f214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:49.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T10:54:49.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:54:49.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T10:54:49.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:54:49.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:54:49.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:49.055 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:54:49.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:54:49.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:49.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:54:49.334 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:49.335+0000 7f145f214640 -1 Falling back to public interface 2024-03-30T10:54:49.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:54:50.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:54:50.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:50.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:54:50.422 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:54:50.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:50.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:54:50.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:54:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:51.540+0000 7f145f214640 -1 osd.1 28 log_to_monitors true 2024-03-30T10:54:51.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:54:51.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:51.729 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:54:51.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:54:51.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:51.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:54:52.043 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 34 up_thru 16 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/1921949759,v1:127.0.0.1:6811/1921949759] [v2:127.0.0.1:6812/1921949759,v1:127.0.0.1:6813/1921949759] exists,up 40300248-f7e0-48fe-bd94-3f130e2aad2c 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-03-30T10:54:52.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28888/$cluster-$name.asok' 2024-03-30T10:54:52.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28888/$cluster-$name.asok' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-03-30T10:54:52.049 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T10:54:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T10:54:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3f31a945-1d20-42c1-9589-211fba072362 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28888/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T10:54:52.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-03-30T10:54:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T10:54:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T10:54:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T10:54:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T10:54:52.082 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:52.083+0000 7fb8a292a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:52.129 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:52.130+0000 7fb8a292a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:52.130 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:52.131+0000 7fb8a292a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:52.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:54:52.505 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:54:52.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:52.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:54:52.647 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:52.647+0000 7fb8a292a640 -1 Falling back to public interface 2024-03-30T10:54:52.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:54:53.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:54:53.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:53.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:54:53.891 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:54:53.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:53.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:54:54.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:54:54.986 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:54:54.987+0000 7fb8a292a640 -1 osd.2 28 log_to_monitors true 2024-03-30T10:54:55.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:54:55.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:54:55.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:54:55.195 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:54:55.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:54:55.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:54:55.499 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 38 up_thru 16 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/328701496,v1:127.0.0.1:6819/328701496] [v2:127.0.0.1:6820/328701496,v1:127.0.0.1:6821/328701496] exists,up 94f587db-db62-4cab-aa70-34d58911e634 2024-03-30T10:54:55.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:54:55.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:54:55.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:54:55.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-03-30T10:54:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-03-30T10:54:56.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-03-30T10:54:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-03-30T10:54:56.612 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:54:56.612 INFO:tasks.workunit.client.0.smithi003.stdout: "version": "19.0.0-1864-g5dc2ea30", 2024-03-30T10:54:56.612 INFO:tasks.workunit.client.0.smithi003.stdout: "release": "squid", 2024-03-30T10:54:56.612 INFO:tasks.workunit.client.0.smithi003.stdout: "release_type": "dev" 2024-03-30T10:54:56.612 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:56.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:56.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-03-30T10:54:56.721 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:54:56.721 INFO:tasks.workunit.client.0.smithi003.stdout: "version": "19.0.0-1864-g5dc2ea30", 2024-03-30T10:54:56.722 INFO:tasks.workunit.client.0.smithi003.stdout: "release": "squid", 2024-03-30T10:54:56.722 INFO:tasks.workunit.client.0.smithi003.stdout: "release_type": "dev" 2024-03-30T10:54:56.722 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:56.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-30T10:54:56.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-03-30T10:54:56.882 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:54:56.882 INFO:tasks.workunit.client.0.smithi003.stdout: "version": "19.0.0-1864-g5dc2ea30", 2024-03-30T10:54:56.882 INFO:tasks.workunit.client.0.smithi003.stdout: "release": "squid", 2024-03-30T10:54:56.882 INFO:tasks.workunit.client.0.smithi003.stdout: "release_type": "dev" 2024-03-30T10:54:56.882 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:56.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "objxxx", 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: "clones": [ 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 2, 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [ 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 1, 2024-03-30T10:54:56.948 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap01" 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 2, 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap02" 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6, 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "overlaps": [] 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 3, 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [ 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 3, 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap13" 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:56.949 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6, 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "overlaps": [] 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 5, 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [ 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 4, 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap24" 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "id": 5, 2024-03-30T10:54:56.950 INFO:tasks.workunit.client.0.smithi003.stdout: "name": "snap25" 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6, 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: "overlaps": [] 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: { 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: "id": "head", 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: "snapshots": [], 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: "size": 6 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: } 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout: ] 2024-03-30T10:54:56.951 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:54:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-03-30T10:54:57.326 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is unset 2024-03-30T10:54:57.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-03-30T10:54:57.820 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is unset 2024-03-30T10:54:57.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-30T10:54:57.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-30T10:54:58.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-03-30T10:54:58.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-30T10:54:58.389 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:54:58.389 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-03-30T10:54:56.592597+0000 26'7 40:67 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-30T10:54:16.211154+0000 26'7 2024-03-30T10:54:16.211154+0000 0 1 no scrub is scheduled 4 0 2024-03-30T10:54:58.389 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-30T10:54:52.940114+0000 0'0 36:26 [0,1] 0 [0,1] 0 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 no scrub is scheduled 0 0 2024-03-30T10:54:58.389 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-30T10:54:52.845252+0000 0'0 36:19 [0,1] 0 [0,1] 0 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:54:58.389 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.701266+0000 0'0 40:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 no scrub is scheduled 0 0 2024-03-30T10:54:58.390 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:54:58.390 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:54:58.390 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:54:58.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-03-30T10:55:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-30T10:55:00.507 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:55:00.507 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": true, 2024-03-30T10:55:00.507 INFO:tasks.workunit.client.0.smithi003.stdout: "must": true, 2024-03-30T10:55:00.508 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "0.000000" 2024-03-30T10:55:00.508 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:55:00.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-03-30T10:55:05.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-30T10:55:05.791 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-03-30T10:55:00.670521+0000 26'7 42:82 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-30T10:55:00.670422+0000 26'7 2024-03-30T10:55:00.670422+0000 0 1 periodic scrub scheduled @ 2024-03-31T10:55:00.670422+0000 4 0 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.257016+0000 0'0 42:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.153389+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.701266+0000 0'0 42:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:55:05.792 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:55:05.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:55:05.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-30T10:55:05.803 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:55:00.669+0000 7f1441341640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-30T10:55:05.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-03-30T10:55:05.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-30T10:55:05.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-03-30T10:55:05.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-03-30T10:55:05.808 INFO:tasks.workunit.client.0.smithi003.stdout:prev count: 1 2024-03-30T10:55:05.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-03-30T10:55:05.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-30T10:55:05.902 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T10:55:05.902 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": true, 2024-03-30T10:55:05.903 INFO:tasks.workunit.client.0.smithi003.stdout: "must": true, 2024-03-30T10:55:05.903 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "0.000000" 2024-03-30T10:55:05.903 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T10:55:05.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-03-30T10:55:10.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-30T10:55:11.188 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2024-03-30T10:55:06.673037+0000 26'7 42:91 [1,2,0] 1 [1,2,0] 1 26'7 2024-03-30T10:55:06.672938+0000 26'7 2024-03-30T10:55:06.672938+0000 0 1 periodic scrub scheduled @ 2024-03-31T10:55:06.672938+0000 4 0 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.257016+0000 0'0 42:42 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.153389+0000 0'0 42:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:54:56.701266+0000 0'0 42:38 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-30T10:54:06.062739+0000 0'0 2024-03-30T10:54:06.062739+0000 0 0 periodic scrub scheduled @ 2024-03-31T10:54:06.062739+0000 0 0 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:55:11.189 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:55:11.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:55:11.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-30T10:55:11.200 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:55:00.669+0000 7f1441341640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-30T10:55:11.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-30T10:55:11.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-03-30T10:55:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-03-30T10:55:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-30T10:55:11.203 INFO:tasks.workunit.client.0.smithi003.stdout:current count: 1 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:55:11.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:55:11.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:55:11.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-30T10:55:11.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-30T10:55:11.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-30T10:55:11.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-03-30T10:55:11.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-30T10:55:12.800 INFO:tasks.workunit.client.0.smithi003.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:55:12.800 INFO:tasks.workunit.client.0.smithi003.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-30T10:55:12.800 INFO:tasks.workunit.client.0.smithi003.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-30T10:55:12.800 INFO:tasks.workunit.client.0.smithi003.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-30T10:55:12.800 INFO:tasks.workunit.client.0.smithi003.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:55:12.800 INFO:tasks.workunit.client.0.smithi003.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-30T10:55:12.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-30T10:55:12.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-30T10:55:12.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-30T10:55:12.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T10:55:13.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-30T10:55:13.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:55:13.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:55:13.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:55:13.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:55:13.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:55:13.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:55:13.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T10:55:13.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T10:55:13.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T10:55:13.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T10:55:13.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T10:55:13.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T10:55:13.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:55:13.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T10:55:13.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T10:55:13.956 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:55:13.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T10:55:13.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T10:55:13.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-30T10:55:13.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T10:55:13.975 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:13.975 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:55:13.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28888 2024-03-30T10:55:13.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T10:55:13.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:55:13.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:55:13.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:55:13.979 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T10:55:13.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T10:55:13.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T10:55:13.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T10:55:13.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T10:55:13.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T10:55:13.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:55:13.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T10:55:13.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T10:55:13.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:55:13.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T10:55:13.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-30T10:55:13.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-30T10:55:13.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T10:55:13.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:13.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28888 2024-03-30T10:55:13.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28888 2024-03-30T10:55:13.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T10:55:13.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T10:55:13.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-30T10:55:13.988 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T10:55:13.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T10:55:14.059 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-03-30T10:55:14.059 DEBUG:teuthology.orchestra.run.smithi003:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-03-30T10:55:14.122 INFO:tasks.workunit.client.0.smithi003.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-30T10:55:14.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-03-30T10:55:14.129 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-03-30T10:55:14.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none ' 2024-03-30T10:55:14.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-03-30T10:55:14.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-03-30T10:55:14.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-30T10:55:14.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-03-30T10:55:14.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_recovery_scrub_2' 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.45029 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-03-30T10:55:14.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T10:55:14.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T10:55:14.136 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T10:55:14.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:55:14.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T10:55:14.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T10:55:14.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-30T10:55:14.139 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T10:55:14.139 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:14.139 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45029 2024-03-30T10:55:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T10:55:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T10:55:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-30T10:55:14.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T10:55:14.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:14.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:14.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.45029 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T10:55:14.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-30T10:55:14.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-30T10:55:14.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T10:55:14.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:55:14.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:55:14.521 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:55:14.521 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:14.521 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:14.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:14.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-30T10:55:14.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T10:55:14.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T10:55:14.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T10:55:14.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T10:55:14.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T10:55:14.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T10:55:14.723 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T10:55:14.723 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T10:55:14.723 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T10:55:14.724 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T10:55:14.724 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:14.724 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:14.724 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45029/ceph-mon.a.asok 2024-03-30T10:55:14.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T10:55:14.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45029/ceph-mon.a.asok config get fsid 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45029/ceph-mon.a.asok 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T10:55:14.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45029/ceph-mon.a.asok config get mon_host 2024-03-30T10:55:14.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-03-30T10:55:14.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-30T10:55:14.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T10:55:14.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T10:55:14.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T10:55:14.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-30T10:55:14.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:15.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T10:55:15.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T10:55:15.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-30T10:55:15.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-30T10:55:15.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-30T10:55:15.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-30T10:55:15.744 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-03-30T10:55:15.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-03-30T10:55:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:55:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-03-30T10:55:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:55:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:55:15.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:55:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=035b1a62-b7ba-4d6d-9442-c4ef5de02304 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 035b1a62-b7ba-4d6d-9442-c4ef5de02304' 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 035b1a62-b7ba-4d6d-9442-c4ef5de02304 2024-03-30T10:55:15.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:55:15.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCT7wdm4ve3LRAARdkfRLM44Al0b6b3AxdpPA== 2024-03-30T10:55:15.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCT7wdm4ve3LRAARdkfRLM44Al0b6b3AxdpPA=="}' 2024-03-30T10:55:15.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 035b1a62-b7ba-4d6d-9442-c4ef5de02304 -i td/osd-recovery-scrub/0/new.json 2024-03-30T10:55:15.970 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:55:15.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-30T10:55:15.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCT7wdm4ve3LRAARdkfRLM44Al0b6b3AxdpPA== --osd-uuid 035b1a62-b7ba-4d6d-9442-c4ef5de02304 2024-03-30T10:55:16.011 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.012+0000 7f020f36d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:16.021 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.021+0000 7f020f36d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:16.022 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.023+0000 7f020f36d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:16.022 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.023+0000 7f020f36d640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-30T10:55:16.022 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.023+0000 7f020f36d640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-30T10:55:16.023 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.023+0000 7f020f36d640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-30T10:55:16.113 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:16.113+0000 7f020f36d640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-30T10:55:21.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-30T10:55:21.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:55:21.529 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T10:55:21.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T10:55:21.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:55:21.902 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T10:55:21.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T10:55:21.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-30T10:55:21.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:55:21.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:55:21.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:55:21.928 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:21.929+0000 7f49008f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:21.930 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:21.931+0000 7f49008f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:21.931 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:21.932+0000 7f49008f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:22.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T10:55:22.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:55:22.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T10:55:22.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:55:22.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:55:22.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:22.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:55:22.186 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:55:22.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:22.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:55:22.444 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:22.445+0000 7f49008f5640 -1 Falling back to public interface 2024-03-30T10:55:22.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:23.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:23.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:23.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:55:23.472 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:55:23.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:23.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:55:23.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:24.762 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:24.762+0000 7f49008f5640 -1 osd.0 0 log_to_monitors true 2024-03-30T10:55:24.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:24.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:24.765 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:55:24.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:55:24.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:24.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:55:25.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:26.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:26.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:26.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:55:26.095 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:55:26.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:26.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:55:26.387 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/32349324,v1:127.0.0.1:6803/32349324] [v2:127.0.0.1:6804/32349324,v1:127.0.0.1:6805/32349324] exists,up 035b1a62-b7ba-4d6d-9442-c4ef5de02304 2024-03-30T10:55:26.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:55:26.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:55:26.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:55:26.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:55:26.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:55:26.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:55:26.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1eeccf5-bfcb-40ee-aca4-30e31676b4fd 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a1eeccf5-bfcb-40ee-aca4-30e31676b4fd' 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 a1eeccf5-bfcb-40ee-aca4-30e31676b4fd 2024-03-30T10:55:26.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:55:26.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCe7wdm7axbGBAAUVV6U7b3WKO55pmOFml9BA== 2024-03-30T10:55:26.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCe7wdm7axbGBAAUVV6U7b3WKO55pmOFml9BA=="}' 2024-03-30T10:55:26.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1eeccf5-bfcb-40ee-aca4-30e31676b4fd -i td/osd-recovery-scrub/1/new.json 2024-03-30T10:55:26.772 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:55:26.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-30T10:55:26.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCe7wdm7axbGBAAUVV6U7b3WKO55pmOFml9BA== --osd-uuid a1eeccf5-bfcb-40ee-aca4-30e31676b4fd 2024-03-30T10:55:26.812 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.812+0000 7f9de1a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:26.856 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.857+0000 7f9de1a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:26.857 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.858+0000 7f9de1a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:26.857 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.858+0000 7f9de1a11640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-30T10:55:26.857 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.858+0000 7f9de1a11640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-30T10:55:26.858 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.858+0000 7f9de1a11640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-30T10:55:26.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:26.933+0000 7f9de1a11640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-30T10:55:32.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-30T10:55:32.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:55:32.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T10:55:32.899 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T10:55:32.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:55:33.269 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T10:55:33.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T10:55:33.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-30T10:55:33.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:55:33.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:55:33.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:55:33.295 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:33.296+0000 7f9ce860d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:33.297 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:33.298+0000 7f9ce860d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:33.298 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:33.299+0000 7f9ce860d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:33.553 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:55:33.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:55:33.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:33.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:55:33.818 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:33.819+0000 7f9ce860d640 -1 Falling back to public interface 2024-03-30T10:55:33.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:34.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:34.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:34.832 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:55:34.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:55:34.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:34.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:55:35.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:36.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:36.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:36.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:55:36.124 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:55:36.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:36.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:55:36.179 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:36.179+0000 7f9ce860d640 -1 osd.1 0 log_to_monitors true 2024-03-30T10:55:36.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:37.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:37.418 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:55:37.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:37.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:55:37.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:37.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:55:37.714 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/735078003,v1:127.0.0.1:6811/735078003] [v2:127.0.0.1:6812/735078003,v1:127.0.0.1:6813/735078003] exists,up a1eeccf5-bfcb-40ee-aca4-30e31676b4fd 2024-03-30T10:55:37.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:55:37.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-30T10:55:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:55:37.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f9cff4f-a9a9-471c-8860-fc1e2857beb9 2024-03-30T10:55:37.720 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 9f9cff4f-a9a9-471c-8860-fc1e2857beb9 2024-03-30T10:55:37.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9f9cff4f-a9a9-471c-8860-fc1e2857beb9' 2024-03-30T10:55:37.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:55:37.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCp7wdmmAHaKxAAaQWCfH6H8vk8RfhfIzXv5g== 2024-03-30T10:55:37.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCp7wdmmAHaKxAAaQWCfH6H8vk8RfhfIzXv5g=="}' 2024-03-30T10:55:37.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f9cff4f-a9a9-471c-8860-fc1e2857beb9 -i td/osd-recovery-scrub/2/new.json 2024-03-30T10:55:38.098 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:55:38.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-30T10:55:38.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCp7wdmmAHaKxAAaQWCfH6H8vk8RfhfIzXv5g== --osd-uuid 9f9cff4f-a9a9-471c-8860-fc1e2857beb9 2024-03-30T10:55:38.138 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.138+0000 7f67eb8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:38.182 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.182+0000 7f67eb8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:38.184 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.183+0000 7f67eb8b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:38.184 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.183+0000 7f67eb8b1640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-30T10:55:38.184 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.183+0000 7f67eb8b1640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-30T10:55:38.184 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.183+0000 7f67eb8b1640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-30T10:55:38.249 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:38.249+0000 7f67eb8b1640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-30T10:55:44.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-30T10:55:44.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:55:44.213 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T10:55:44.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T10:55:44.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:55:44.580 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T10:55:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T10:55:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-30T10:55:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:55:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:55:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:55:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:44.607+0000 7fd682624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:44.609 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:44.609+0000 7fd682624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:44.610 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:44.611+0000 7fd682624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:44.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:55:44.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:44.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:55:45.139 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:45.139+0000 7fd682624640 -1 Falling back to public interface 2024-03-30T10:55:45.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:46.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:46.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:46.163 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:55:46.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:55:46.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:46.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:55:46.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:47.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:47.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:47.456 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:55:47.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:55:47.457 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:47.457+0000 7fd682624640 -1 osd.2 0 log_to_monitors true 2024-03-30T10:55:47.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:47.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:55:47.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:48.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:48.751 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:55:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:55:48.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:48.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:55:49.040 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3888035239,v1:127.0.0.1:6819/3888035239] [v2:127.0.0.1:6820/3888035239,v1:127.0.0.1:6821/3888035239] exists,up 9f9cff4f-a9a9-471c-8860-fc1e2857beb9 2024-03-30T10:55:49.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:55:49.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:49.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57167aa8-af67-4ba9-b285-ed2382f77ba6 2024-03-30T10:55:49.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 57167aa8-af67-4ba9-b285-ed2382f77ba6' 2024-03-30T10:55:49.048 INFO:tasks.workunit.client.0.smithi003.stdout:add osd3 57167aa8-af67-4ba9-b285-ed2382f77ba6 2024-03-30T10:55:49.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:55:49.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC17wdmvna0AxAAgOJy+mClhJPKJwahswWxnQ== 2024-03-30T10:55:49.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC17wdmvna0AxAAgOJy+mClhJPKJwahswWxnQ=="}' 2024-03-30T10:55:49.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57167aa8-af67-4ba9-b285-ed2382f77ba6 -i td/osd-recovery-scrub/3/new.json 2024-03-30T10:55:49.427 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:55:49.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-30T10:55:49.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQC17wdmvna0AxAAgOJy+mClhJPKJwahswWxnQ== --osd-uuid 57167aa8-af67-4ba9-b285-ed2382f77ba6 2024-03-30T10:55:49.467 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.467+0000 7f600f487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:49.503 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.503+0000 7f600f487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:49.503 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.504+0000 7f600f487640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:49.504 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.505+0000 7f600f487640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-30T10:55:49.504 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.505+0000 7f600f487640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-30T10:55:49.504 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.505+0000 7f600f487640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-30T10:55:49.569 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:49.570+0000 7f600f487640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-30T10:55:55.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-30T10:55:55.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:55:55.343 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd3 key to auth repository 2024-03-30T10:55:55.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-30T10:55:55.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:55:55.708 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.3 2024-03-30T10:55:55.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-30T10:55:55.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-30T10:55:55.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:55:55.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:55:55.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:55:55.734 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:55.735+0000 7fa2a5cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:55.736 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:55.737+0000 7fa2a5cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:55.738 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:55.738+0000 7fa2a5cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:55:55.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-30T10:55:55.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:55:55.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T10:55:55.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:55:55.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:55:55.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:55.991 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:55:55.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:55:55.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:55.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:55:56.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:56.278 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:56.279+0000 7fa2a5cc8640 -1 Falling back to public interface 2024-03-30T10:55:57.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:57.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:57.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:55:57.276 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:55:57.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:57.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:55:57.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:58.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:58.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:58.567 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:55:58.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:55:58.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:58.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:55:58.621 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:55:58.622+0000 7fa2a5cc8640 -1 osd.3 0 log_to_monitors true 2024-03-30T10:55:58.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:55:59.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:55:59.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:55:59.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:55:59.862 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:55:59.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:55:59.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:56:00.173 INFO:tasks.workunit.client.0.smithi003.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1299919171,v1:127.0.0.1:6827/1299919171] [v2:127.0.0.1:6828/1299919171,v1:127.0.0.1:6829/1299919171] exists,up 57167aa8-af67-4ba9-b285-ed2382f77ba6 2024-03-30T10:56:00.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:56:00.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:56:00.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:56:00.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-03-30T10:56:00.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-03-30T10:56:00.550 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'test' created 2024-03-30T10:56:00.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T10:56:01.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-03-30T10:56:01.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T10:56:01.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T10:56:01.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T10:56:01.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T10:56:01.564 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T10:56:01.564 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T10:56:01.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T10:56:01.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T10:56:01.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T10:56:01.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T10:56:01.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T10:56:01.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T10:56:01.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T10:56:01.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T10:56:01.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T10:56:02.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T10:56:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-03-30T10:56:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-03-30T10:56:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-03-30T10:56:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T10:56:02.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T10:56:02.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-30T10:56:02.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-30T10:56:02.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-03-30T10:56:02.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T10:56:02.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T10:56:02.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-30T10:56:02.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-30T10:56:02.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444' 2024-03-30T10:56:02.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T10:56:02.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T10:56:02.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-03-30T10:56:02.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-03-30T10:56:02.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509444 3-85899345922' 2024-03-30T10:56:02.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T10:56:02.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-03-30T10:56:02.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T10:56:02.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T10:56:02.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-03-30T10:56:02.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T10:56:02.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-03-30T10:56:02.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-03-30T10:56:02.448 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836489 2024-03-30T10:56:02.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T10:56:02.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-03-30T10:56:02.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T10:56:03.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T10:56:03.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T10:56:04.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-03-30T10:56:04.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T10:56:04.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-30T10:56:04.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T10:56:04.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T10:56:04.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-30T10:56:04.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T10:56:04.030 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672966 2024-03-30T10:56:04.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-30T10:56:04.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-30T10:56:04.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T10:56:04.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-03-30T10:56:04.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T10:56:04.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-30T10:56:04.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T10:56:04.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T10:56:04.317 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-30T10:56:04.317 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T10:56:04.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-30T10:56:04.318 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509444 2024-03-30T10:56:04.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-30T10:56:04.319 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T10:56:04.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-30T10:56:04.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T10:56:04.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-03-30T10:56:04.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T10:56:04.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T10:56:04.602 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-03-30T10:56:04.603 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T10:56:04.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-03-30T10:56:04.604 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345922 2024-03-30T10:56:04.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-03-30T10:56:04.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T10:56:04.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-03-30T10:56:04.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T10:56:04.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T10:56:04.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T10:56:05.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T10:56:05.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T10:56:05.245 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T10:56:05.245 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T10:56:05.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T10:56:05.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T10:56:05.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T10:56:05.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T10:56:05.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T10:56:05.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T10:56:05.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T10:56:05.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T10:56:06.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T10:56:06.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T10:56:06.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T10:56:06.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-03-30T10:56:06.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-03-30T10:56:06.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-03-30T10:56:06.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-03-30T10:56:06.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-30T10:56:06.572 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:56:06.572 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T10:56:01.514088+0000 0'0 23:8 [1] 1 [1] 1 0'0 2024-03-30T10:56:00.500896+0000 0'0 2024-03-30T10:56:00.500896+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:34:47.870470+0000 0 0 2024-03-30T10:56:06.572 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:56:06.572 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:56:06.573 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:56:06.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.45029 bs=1M count=50 2024-03-30T10:56:06.721 INFO:tasks.workunit.client.0.smithi003.stderr:50+0 records in 2024-03-30T10:56:06.722 INFO:tasks.workunit.client.0.smithi003.stderr:50+0 records out 2024-03-30T10:56:06.722 INFO:tasks.workunit.client.0.smithi003.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.137684 s, 381 MB/s 2024-03-30T10:56:06.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-03-30T10:56:06.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:06.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.45029 2024-03-30T10:56:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.45029 2024-03-30T10:56:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.45029 2024-03-30T10:56:11.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:11.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.45029 2024-03-30T10:56:12.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:12.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.45029 2024-03-30T10:56:14.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:14.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.45029 2024-03-30T10:56:15.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:15.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.45029 2024-03-30T10:56:16.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:16.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.45029 2024-03-30T10:56:18.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:18.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.45029 2024-03-30T10:56:19.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:19.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.45029 2024-03-30T10:56:21.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:21.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.45029 2024-03-30T10:56:22.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:22.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.45029 2024-03-30T10:56:24.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:24.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.45029 2024-03-30T10:56:25.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:25.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.45029 2024-03-30T10:56:27.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:27.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.45029 2024-03-30T10:56:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:28.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.45029 2024-03-30T10:56:30.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:30.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.45029 2024-03-30T10:56:31.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:31.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.45029 2024-03-30T10:56:33.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:33.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.45029 2024-03-30T10:56:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.45029 2024-03-30T10:56:36.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:36.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.45029 2024-03-30T10:56:37.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:37.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.45029 2024-03-30T10:56:39.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:39.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.45029 2024-03-30T10:56:40.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:40.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.45029 2024-03-30T10:56:42.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:42.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.45029 2024-03-30T10:56:43.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:43.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.45029 2024-03-30T10:56:45.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:45.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.45029 2024-03-30T10:56:46.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:46.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.45029 2024-03-30T10:56:48.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:48.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.45029 2024-03-30T10:56:49.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:49.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.45029 2024-03-30T10:56:51.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:51.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.45029 2024-03-30T10:56:52.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:52.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.45029 2024-03-30T10:56:54.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:54.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.45029 2024-03-30T10:56:55.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:55.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.45029 2024-03-30T10:56:56.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:56.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.45029 2024-03-30T10:56:58.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:58.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.45029 2024-03-30T10:56:59.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:56:59.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.45029 2024-03-30T10:57:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.45029 2024-03-30T10:57:02.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:02.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.45029 2024-03-30T10:57:04.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:04.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.45029 2024-03-30T10:57:05.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:05.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.45029 2024-03-30T10:57:07.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:07.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.45029 2024-03-30T10:57:08.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:08.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.45029 2024-03-30T10:57:10.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:10.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.45029 2024-03-30T10:57:11.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:11.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.45029 2024-03-30T10:57:13.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:13.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.45029 2024-03-30T10:57:14.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:14.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.45029 2024-03-30T10:57:16.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:16.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.45029 2024-03-30T10:57:17.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:17.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.45029 2024-03-30T10:57:19.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:19.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.45029 2024-03-30T10:57:20.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:20.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.45029 2024-03-30T10:57:21.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:21.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.45029 2024-03-30T10:57:23.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:23.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.45029 2024-03-30T10:57:24.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:24.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.45029 2024-03-30T10:57:26.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:26.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.45029 2024-03-30T10:57:27.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:27.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.45029 2024-03-30T10:57:29.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:29.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.45029 2024-03-30T10:57:30.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:30.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.45029 2024-03-30T10:57:32.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:32.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.45029 2024-03-30T10:57:33.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:33.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.45029 2024-03-30T10:57:35.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:35.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.45029 2024-03-30T10:57:36.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:36.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.45029 2024-03-30T10:57:38.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:38.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.45029 2024-03-30T10:57:39.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:39.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.45029 2024-03-30T10:57:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:41.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.45029 2024-03-30T10:57:42.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:42.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.45029 2024-03-30T10:57:44.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:44.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.45029 2024-03-30T10:57:45.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:45.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.45029 2024-03-30T10:57:47.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:47.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.45029 2024-03-30T10:57:48.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:48.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.45029 2024-03-30T10:57:50.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:50.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.45029 2024-03-30T10:57:51.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:51.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.45029 2024-03-30T10:57:53.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:53.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.45029 2024-03-30T10:57:54.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:54.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.45029 2024-03-30T10:57:56.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:56.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.45029 2024-03-30T10:57:57.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:57.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.45029 2024-03-30T10:57:59.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:57:59.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.45029 2024-03-30T10:58:00.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:00.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.45029 2024-03-30T10:58:01.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:01.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.45029 2024-03-30T10:58:03.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:03.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.45029 2024-03-30T10:58:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.45029 2024-03-30T10:58:06.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:06.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.45029 2024-03-30T10:58:07.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:07.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.45029 2024-03-30T10:58:09.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:09.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.45029 2024-03-30T10:58:10.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:10.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.45029 2024-03-30T10:58:12.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:12.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.45029 2024-03-30T10:58:13.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:13.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.45029 2024-03-30T10:58:15.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:15.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.45029 2024-03-30T10:58:16.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:16.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.45029 2024-03-30T10:58:18.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:18.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.45029 2024-03-30T10:58:19.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:19.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.45029 2024-03-30T10:58:21.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:21.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.45029 2024-03-30T10:58:22.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:22.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.45029 2024-03-30T10:58:24.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:24.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.45029 2024-03-30T10:58:25.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:25.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.45029 2024-03-30T10:58:27.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:27.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.45029 2024-03-30T10:58:28.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:28.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.45029 2024-03-30T10:58:30.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:30.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.45029 2024-03-30T10:58:31.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:31.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.45029 2024-03-30T10:58:33.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-30T10:58:33.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.45029 2024-03-30T10:58:34.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.45029 2024-03-30T10:58:34.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-03-30T10:58:34.980 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 4 2024-03-30T10:58:34.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-03-30T10:58:34.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-03-30T10:58:34.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-30T10:58:34.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-30T10:58:34.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-30T10:58:34.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-30T10:58:35.261 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:58:35.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-30T10:58:37.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-03-30T10:58:37.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-03-30T10:58:37.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-03-30T10:58:37.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-30T10:58:37.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-30T10:58:37.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-30T10:58:37.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-30T10:58:37.554 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:58:37.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-30T10:58:39.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-03-30T10:58:39.567 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-03-30T10:58:39.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-03-30T10:58:39.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-30T10:58:39.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-30T10:58:39.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-30T10:58:39.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-30T10:58:39.920 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:58:39.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-03-30T10:58:39.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-03-30T10:58:39.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-30T10:58:40.290 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T10:58:40.291 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-03-30T10:58:36.717731+0000 24'1300 27:1323 [1,0,2,3] 1 [1,0] 1 0'0 2024-03-30T10:56:00.500896+0000 0'0 2024-03-30T10:56:00.500896+0000 0 0 no scrub is scheduled 0 0 2024-03-30T10:58:40.291 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T10:58:40.291 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T10:58:40.291 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T10:58:40.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-03-30T10:58:50.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-03-30T10:58:50.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:58:50.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:58:50.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:58:50.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:58:50.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:58:55.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:58:55.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-03-30T10:58:55.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-03-30T10:58:55.620 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-03-30T10:58:55.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-03-30T10:58:55.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-30T10:58:55.624 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:42.444+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.624 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:43.400+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.624 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:44.445+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.625 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:46.450+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.625 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:50.347+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.625 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:51.372+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.625 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:54.398+0000 7f48f9898640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-30T10:58:55.636 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:36.726+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.636 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:37.762+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:42.746+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:44.787+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:45.746+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:49.622+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:50.661+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:53.635+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:54.631+0000 7f9ce0f66640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-30T10:58:55.639 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:38.594+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.639 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:39.620+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.639 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:40.645+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.639 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:41.681+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.639 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:45.662+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.639 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:47.651+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.640 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:51.728+0000 7fd67af08640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:38.297+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:39.306+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:40.286+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:43.323+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:44.346+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:48.285+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:50.245+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:51.219+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T10:58:54.191+0000 7fa29ec6b640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-30T10:58:55.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-03-30T10:58:55.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-03-30T10:58:55.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-03-30T10:58:55.643 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-03-30T10:58:55.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-03-30T10:58:55.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-30T10:58:55.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-30T10:58:55.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-03-30T10:58:55.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-03-30T10:58:55.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-30T10:58:55.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-30T10:58:55.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-03-30T10:58:55.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-03-30T10:58:55.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-30T10:58:55.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-30T10:58:55.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-03-30T10:58:55.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-03-30T10:58:55.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:55.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-30T10:58:55.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-30T10:58:55.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T10:58:55.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-30T10:58:55.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:58:55.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:58:55.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:58:55.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:58:55.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:58:55.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:58:55.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T10:58:55.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T10:58:55.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T10:58:55.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T10:58:55.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T10:58:55.678 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T10:58:55.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:58:55.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T10:58:55.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T10:58:55.679 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:58:55.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T10:58:55.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T10:58:55.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-30T10:58:55.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T10:58:55.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:55.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:55.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45029 2024-03-30T10:58:55.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T10:58:55.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T10:58:55.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-03-30T10:58:55.704 INFO:tasks.workunit.client.0.smithi003.stdout:TEST PASSED 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.45029 2024-03-30T10:58:55.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T10:58:55.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T10:58:55.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T10:58:55.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T10:58:55.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T10:58:55.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T10:58:55.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T10:58:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-30T10:58:55.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T10:58:55.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:55.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:55.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45029 2024-03-30T10:58:55.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T10:58:55.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T10:58:55.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-30T10:58:55.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T10:58:55.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:55.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:55.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.45029 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-30T10:58:55.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:58:56.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-30T10:58:56.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T10:58:56.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T10:58:56.309 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T10:58:56.310 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T10:58:56.310 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:56.310 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:56.310 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45029/ceph-mon.a.asok 2024-03-30T10:58:56.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T10:58:56.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45029/ceph-mon.a.asok config get fsid 2024-03-30T10:58:56.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T10:58:56.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T10:58:56.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T10:58:56.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T10:58:56.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.45029/ceph-mon.a.asok 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T10:58:56.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.45029/ceph-mon.a.asok config get mon_host 2024-03-30T10:58:56.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-03-30T10:58:56.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-30T10:58:56.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T10:58:56.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T10:58:56.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T10:58:56.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-30T10:58:56.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:57.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:58:57.302 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T10:58:57.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T10:58:57.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-30T10:58:57.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-30T10:58:57.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-30T10:58:57.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-30T10:58:57.331 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-03-30T10:58:57.332 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-03-30T10:58:57.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:58:57.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:58:57.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:58:57.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:58:57.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:58:57.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:58:57.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:58:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:58:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T10:58:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-30T10:58:57.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:58:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55aa1b54-2c94-430b-8cab-626c12f6b7e8 2024-03-30T10:58:57.343 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 55aa1b54-2c94-430b-8cab-626c12f6b7e8 2024-03-30T10:58:57.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 55aa1b54-2c94-430b-8cab-626c12f6b7e8' 2024-03-30T10:58:57.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:58:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBx8AdmfokaFRAAbSPaDhziWwzSaJRIYEGvmQ== 2024-03-30T10:58:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBx8AdmfokaFRAAbSPaDhziWwzSaJRIYEGvmQ=="}' 2024-03-30T10:58:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55aa1b54-2c94-430b-8cab-626c12f6b7e8 -i td/osd-recovery-scrub/0/new.json 2024-03-30T10:58:57.563 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:58:57.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-30T10:58:57.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBx8AdmfokaFRAAbSPaDhziWwzSaJRIYEGvmQ== --osd-uuid 55aa1b54-2c94-430b-8cab-626c12f6b7e8 2024-03-30T10:58:57.606 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.607+0000 7fed09c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:58:57.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.615+0000 7fed09c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:58:57.615 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.616+0000 7fed09c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:58:57.615 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.616+0000 7fed09c68640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-30T10:58:57.616 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.616+0000 7fed09c68640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-30T10:58:57.616 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.616+0000 7fed09c68640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-30T10:58:57.689 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:58:57.690+0000 7fed09c68640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-30T10:59:03.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-30T10:59:03.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:59:03.256 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T10:59:03.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T10:59:03.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:59:03.675 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T10:59:03.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T10:59:03.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:03.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:59:03.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:59:03.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:59:03.706 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:03.707+0000 7f304c549640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:03.709 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:03.710+0000 7f304c549640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:03.710 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:03.711+0000 7f304c549640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:04.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T10:59:04.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:59:04.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T10:59:04.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:59:04.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:59:04.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:04.001 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:59:04.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:59:04.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:04.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:59:04.234 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:04.234+0000 7f304c549640 -1 Falling back to public interface 2024-03-30T10:59:04.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:05.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:05.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:05.286 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:59:05.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:59:05.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:05.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:59:05.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:06.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:06.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:06.576 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:59:06.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:59:06.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:06.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:59:06.581 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:06.580+0000 7f304c549640 -1 osd.0 0 log_to_monitors true 2024-03-30T10:59:06.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:07.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:07.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:07.921 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:59:07.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:59:07.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:07.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T10:59:08.218 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/511471325,v1:127.0.0.1:6803/511471325] [v2:127.0.0.1:6804/511471325,v1:127.0.0.1:6805/511471325] exists,up 55aa1b54-2c94-430b-8cab-626c12f6b7e8 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:59:08.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:59:08.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=064434d5-90b2-4f09-bc6e-02795bd9329d 2024-03-30T10:59:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 064434d5-90b2-4f09-bc6e-02795bd9329d' 2024-03-30T10:59:08.225 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 064434d5-90b2-4f09-bc6e-02795bd9329d 2024-03-30T10:59:08.225 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:59:08.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB88AdmTi1YDhAAAsbnH2L3kFZk6fHx5BbvAQ== 2024-03-30T10:59:08.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB88AdmTi1YDhAAAsbnH2L3kFZk6fHx5BbvAQ=="}' 2024-03-30T10:59:08.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 064434d5-90b2-4f09-bc6e-02795bd9329d -i td/osd-recovery-scrub/1/new.json 2024-03-30T10:59:08.606 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:59:08.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-30T10:59:08.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB88AdmTi1YDhAAAsbnH2L3kFZk6fHx5BbvAQ== --osd-uuid 064434d5-90b2-4f09-bc6e-02795bd9329d 2024-03-30T10:59:08.650 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.651+0000 7f891840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:08.691 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.691+0000 7f891840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:08.692 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.692+0000 7f891840d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:08.692 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.693+0000 7f891840d640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-30T10:59:08.692 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.693+0000 7f891840d640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-30T10:59:08.692 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.693+0000 7f891840d640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-30T10:59:08.757 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:08.758+0000 7f891840d640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-30T10:59:14.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-30T10:59:14.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:59:14.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T10:59:14.740 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T10:59:14.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:59:15.110 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T10:59:15.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T10:59:15.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:15.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:59:15.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:59:15.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:59:15.140 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:15.141+0000 7f1a0541c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:15.142 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:15.143+0000 7f1a0541c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:15.143 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:15.144+0000 7f1a0541c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:59:15.395 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:59:15.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:15.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:59:15.657 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:15.658+0000 7f1a0541c640 -1 Falling back to public interface 2024-03-30T10:59:15.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:16.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:16.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:16.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:59:16.679 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:59:16.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:16.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:59:16.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:17.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:17.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:17.975 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:59:17.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:59:17.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:17.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:59:18.005 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:18.004+0000 7f1a0541c640 -1 osd.1 0 log_to_monitors true 2024-03-30T10:59:18.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:19.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:19.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:19.322 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:59:19.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:59:19.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:19.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T10:59:19.612 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1005493427,v1:127.0.0.1:6811/1005493427] [v2:127.0.0.1:6812/1005493427,v1:127.0.0.1:6813/1005493427] exists,up 064434d5-90b2-4f09-bc6e-02795bd9329d 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T10:59:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:59:19.620 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:59:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:59:19.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:59:19.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T10:59:19.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-30T10:59:19.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:59:19.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53d1d1ab-93f4-4c5d-b1d0-df79707866b1 2024-03-30T10:59:19.622 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 53d1d1ab-93f4-4c5d-b1d0-df79707866b1 2024-03-30T10:59:19.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 53d1d1ab-93f4-4c5d-b1d0-df79707866b1' 2024-03-30T10:59:19.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:59:19.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCH8AdmK77FJRAAhYJutchrQJ8cXICkQc+asg== 2024-03-30T10:59:19.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCH8AdmK77FJRAAhYJutchrQJ8cXICkQc+asg=="}' 2024-03-30T10:59:19.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53d1d1ab-93f4-4c5d-b1d0-df79707866b1 -i td/osd-recovery-scrub/2/new.json 2024-03-30T10:59:19.982 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:59:19.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-30T10:59:19.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCH8AdmK77FJRAAhYJutchrQJ8cXICkQc+asg== --osd-uuid 53d1d1ab-93f4-4c5d-b1d0-df79707866b1 2024-03-30T10:59:20.026 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.025+0000 7f49822c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:20.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.057+0000 7f49822c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:20.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.058+0000 7f49822c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:20.059 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.059+0000 7f49822c4640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-30T10:59:20.059 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.059+0000 7f49822c4640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-30T10:59:20.059 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.059+0000 7f49822c4640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-30T10:59:20.132 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:20.132+0000 7f49822c4640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-30T10:59:26.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-30T10:59:26.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:59:26.175 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T10:59:26.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T10:59:26.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:59:26.547 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T10:59:26.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T10:59:26.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:26.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:59:26.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:59:26.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:59:26.577 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:26.578+0000 7f5afacbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:26.579 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:26.580+0000 7f5afacbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:26.580 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:26.581+0000 7f5afacbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:59:26.838 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:59:26.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:26.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:59:27.100 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:27.101+0000 7f5afacbd640 -1 Falling back to public interface 2024-03-30T10:59:27.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:28.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:28.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:28.118 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:59:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:59:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:59:28.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:29.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:29.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:29.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:59:29.427 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:59:29.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:29.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:59:29.430 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:29.431+0000 7f5afacbd640 -1 osd.2 0 log_to_monitors true 2024-03-30T10:59:29.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:30.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:30.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:30.731 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:59:30.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:59:30.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:30.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T10:59:31.023 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3057518830,v1:127.0.0.1:6819/3057518830] [v2:127.0.0.1:6820/3057518830,v1:127.0.0.1:6821/3057518830] exists,up 53d1d1ab-93f4-4c5d-b1d0-df79707866b1 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:59:31.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:59:31.029 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:59:31.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1bb3041f-b1ec-4bc9-91e6-abbe1a33e662 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1bb3041f-b1ec-4bc9-91e6-abbe1a33e662' 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stdout:add osd3 1bb3041f-b1ec-4bc9-91e6-abbe1a33e662 2024-03-30T10:59:31.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:59:31.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCT8AdmECy3AhAAvvg79YjAaIOjq7KcsaLWUA== 2024-03-30T10:59:31.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCT8AdmECy3AhAAvvg79YjAaIOjq7KcsaLWUA=="}' 2024-03-30T10:59:31.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1bb3041f-b1ec-4bc9-91e6-abbe1a33e662 -i td/osd-recovery-scrub/3/new.json 2024-03-30T10:59:31.408 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:59:31.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-30T10:59:31.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCT8AdmECy3AhAAvvg79YjAaIOjq7KcsaLWUA== --osd-uuid 1bb3041f-b1ec-4bc9-91e6-abbe1a33e662 2024-03-30T10:59:31.451 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.452+0000 7f7217814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:31.493 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.494+0000 7f7217814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:31.493 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.494+0000 7f7217814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:31.494 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.495+0000 7f7217814640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-30T10:59:31.494 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.495+0000 7f7217814640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-30T10:59:31.494 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.495+0000 7f7217814640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-30T10:59:31.568 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:31.568+0000 7f7217814640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-30T10:59:37.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-30T10:59:37.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:59:37.365 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd3 key to auth repository 2024-03-30T10:59:37.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-30T10:59:37.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:59:37.732 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.3 2024-03-30T10:59:37.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-30T10:59:37.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:37.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:59:37.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:59:37.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:59:37.762 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:37.763+0000 7f85d3719640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:37.764 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:37.765+0000 7f85d3719640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:37.765 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:37.766+0000 7f85d3719640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:59:38.016 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:59:38.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:38.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:59:38.283 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:38.284+0000 7f85d3719640 -1 Falling back to public interface 2024-03-30T10:59:38.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:39.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:39.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:39.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:59:39.298 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:59:39.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:39.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:59:39.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:40.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:40.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:40.592 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:59:40.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:59:40.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:40.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:59:40.723 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:40.724+0000 7f85d3719640 -1 osd.3 0 log_to_monitors true 2024-03-30T10:59:40.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:41.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:41.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:41.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:59:41.907 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:59:41.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:41.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T10:59:42.222 INFO:tasks.workunit.client.0.smithi003.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3135449086,v1:127.0.0.1:6827/3135449086] [v2:127.0.0.1:6828/3135449086,v1:127.0.0.1:6829/3135449086] exists,up 1bb3041f-b1ec-4bc9-91e6-abbe1a33e662 2024-03-30T10:59:42.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:59:42.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:59:42.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:42.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=620101ae-50b9-4de3-948b-4c674017c9db 2024-03-30T10:59:42.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 620101ae-50b9-4de3-948b-4c674017c9db' 2024-03-30T10:59:42.230 INFO:tasks.workunit.client.0.smithi003.stdout:add osd4 620101ae-50b9-4de3-948b-4c674017c9db 2024-03-30T10:59:42.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:59:42.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCe8AdmABGLDhAAOVtIF/uUjQ8nESjAAW8XtQ== 2024-03-30T10:59:42.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCe8AdmABGLDhAAOVtIF/uUjQ8nESjAAW8XtQ=="}' 2024-03-30T10:59:42.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 620101ae-50b9-4de3-948b-4c674017c9db -i td/osd-recovery-scrub/4/new.json 2024-03-30T10:59:42.611 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T10:59:42.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-03-30T10:59:42.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCe8AdmABGLDhAAOVtIF/uUjQ8nESjAAW8XtQ== --osd-uuid 620101ae-50b9-4de3-948b-4c674017c9db 2024-03-30T10:59:42.654 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.655+0000 7f82135ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:42.695 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.696+0000 7f82135ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:42.696 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.697+0000 7f82135ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:42.696 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.698+0000 7f82135ac640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-30T10:59:42.697 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.698+0000 7f82135ac640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-30T10:59:42.697 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.698+0000 7f82135ac640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-30T10:59:42.770 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:42.771+0000 7f82135ac640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-03-30T10:59:48.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-03-30T10:59:48.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:59:48.541 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd4 key to auth repository 2024-03-30T10:59:48.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-30T10:59:48.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T10:59:48.906 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.4 2024-03-30T10:59:48.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-30T10:59:48.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:48.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T10:59:48.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T10:59:48.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T10:59:48.937 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:48.938+0000 7f5d72e37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:48.938 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:48.939+0000 7f5d72e37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:48.939 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:48.940+0000 7f5d72e37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:49.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-30T10:59:49.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T10:59:49.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-30T10:59:49.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T10:59:49.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T10:59:49.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:49.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T10:59:49.188 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T10:59:49.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:49.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T10:59:49.468 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:49.469+0000 7f5d72e37640 -1 Falling back to public interface 2024-03-30T10:59:49.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:50.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:50.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:50.473 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T10:59:50.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T10:59:50.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:50.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T10:59:50.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:51.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:51.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:51.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T10:59:51.763 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T10:59:51.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:51.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T10:59:51.853 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:51.854+0000 7f5d72e37640 -1 osd.4 0 log_to_monitors true 2024-03-30T10:59:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T10:59:53.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T10:59:53.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T10:59:53.063 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T10:59:53.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T10:59:53.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T10:59:53.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T10:59:53.392 INFO:tasks.workunit.client.0.smithi003.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1959139981,v1:127.0.0.1:6835/1959139981] [v2:127.0.0.1:6836/1959139981,v1:127.0.0.1:6837/1959139981] exists,up 620101ae-50b9-4de3-948b-4c674017c9db 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T10:59:53.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T10:59:53.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T10:59:53.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89833ddd-c457-45b3-8474-d38bd76a7cfc 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stdout:add osd5 89833ddd-c457-45b3-8474-d38bd76a7cfc 2024-03-30T10:59:53.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 89833ddd-c457-45b3-8474-d38bd76a7cfc' 2024-03-30T10:59:53.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T10:59:53.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCp8Adm42etGBAAlsh0lRVXr+X6pghKTb0AKQ== 2024-03-30T10:59:53.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCp8Adm42etGBAAlsh0lRVXr+X6pghKTb0AKQ=="}' 2024-03-30T10:59:53.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89833ddd-c457-45b3-8474-d38bd76a7cfc -i td/osd-recovery-scrub/5/new.json 2024-03-30T10:59:53.814 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T10:59:53.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-03-30T10:59:53.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCp8Adm42etGBAAlsh0lRVXr+X6pghKTb0AKQ== --osd-uuid 89833ddd-c457-45b3-8474-d38bd76a7cfc 2024-03-30T10:59:53.857 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.859+0000 7f18f3e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:53.864 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.866+0000 7f18f3e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:53.865 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.867+0000 7f18f3e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T10:59:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.867+0000 7f18f3e19640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-30T10:59:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.867+0000 7f18f3e19640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-30T10:59:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.867+0000 7f18f3e19640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-30T10:59:53.965 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T10:59:53.966+0000 7f18f3e19640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-03-30T10:59:59.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-03-30T10:59:59.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T10:59:59.859 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd5 key to auth repository 2024-03-30T10:59:59.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-30T10:59:59.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:00:00.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-30T11:00:00.227 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.5 2024-03-30T11:00:00.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T11:00:00.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:00:00.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:00:00.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:00:00.258 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:00.259+0000 7f227b1bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:00.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:00.261+0000 7f227b1bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:00.261 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:00.262+0000 7f227b1bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:00.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:00:00.523 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:00:00.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:00.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:00:00.785 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:00.786+0000 7f227b1bd640 -1 Falling back to public interface 2024-03-30T11:00:00.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:00:01.804 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:00:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:00:02.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:03.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:03.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:03.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:00:03.100 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:00:03.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:03.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:00:03.128 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:03.129+0000 7f227b1bd640 -1 osd.5 0 log_to_monitors true 2024-03-30T11:00:03.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:04.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:04.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:04.490 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:00:04.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:00:04.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:04.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:00:04.779 INFO:tasks.workunit.client.0.smithi003.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3258251658,v1:127.0.0.1:6843/3258251658] [v2:127.0.0.1:6844/3258251658,v1:127.0.0.1:6845/3258251658] exists,up 89833ddd-c457-45b3-8474-d38bd76a7cfc 2024-03-30T11:00:04.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:00:04.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:00:04.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:00:04.785 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T11:00:04.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-03-30T11:00:04.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:00:04.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=101851ba-711b-4df1-ab45-a218ac61f0db 2024-03-30T11:00:04.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 101851ba-711b-4df1-ab45-a218ac61f0db' 2024-03-30T11:00:04.787 INFO:tasks.workunit.client.0.smithi003.stdout:add osd6 101851ba-711b-4df1-ab45-a218ac61f0db 2024-03-30T11:00:04.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:00:04.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC08Adm6C++LxAA8oYl3YMNqm4LSbY4TpNZLQ== 2024-03-30T11:00:04.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC08Adm6C++LxAA8oYl3YMNqm4LSbY4TpNZLQ=="}' 2024-03-30T11:00:04.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 101851ba-711b-4df1-ab45-a218ac61f0db -i td/osd-recovery-scrub/6/new.json 2024-03-30T11:00:05.151 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:00:05.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-03-30T11:00:05.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC08Adm6C++LxAA8oYl3YMNqm4LSbY4TpNZLQ== --osd-uuid 101851ba-711b-4df1-ab45-a218ac61f0db 2024-03-30T11:00:05.195 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.196+0000 7fdb6cfbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:05.235 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.237+0000 7fdb6cfbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:05.236 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.238+0000 7fdb6cfbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:05.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.238+0000 7fdb6cfbd640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-30T11:00:05.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.238+0000 7fdb6cfbd640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-30T11:00:05.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.238+0000 7fdb6cfbd640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-30T11:00:05.310 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:05.311+0000 7fdb6cfbd640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-03-30T11:00:11.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-03-30T11:00:11.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:00:11.587 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd6 key to auth repository 2024-03-30T11:00:11.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-30T11:00:11.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:00:11.954 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.6 2024-03-30T11:00:11.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-30T11:00:11.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T11:00:11.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:00:11.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:00:11.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:00:11.984 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:11.986+0000 7f0ec640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:11.986 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:11.988+0000 7f0ec640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:11.987 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:11.989+0000 7f0ec640e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:12.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:00:12.239 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:00:12.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:12.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:00:12.508 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:12.510+0000 7f0ec640e640 -1 Falling back to public interface 2024-03-30T11:00:12.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:13.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:13.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:13.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:00:13.544 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:00:13.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:13.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:00:13.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:14.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:14.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:14.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:00:14.837 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:00:14.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:14.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:00:14.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:14.933+0000 7f0ec640e640 -1 osd.6 0 log_to_monitors true 2024-03-30T11:00:15.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:16.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:16.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:16.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:00:16.132 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:00:16.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:16.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:00:16.556 INFO:tasks.workunit.client.0.smithi003.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3538977580,v1:127.0.0.1:6851/3538977580] [v2:127.0.0.1:6852/3538977580,v1:127.0.0.1:6853/3538977580] exists,up 101851ba-711b-4df1-ab45-a218ac61f0db 2024-03-30T11:00:16.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:00:16.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:00:16.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:00:16.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:00:16.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:00:16.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-30T11:00:16.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e5f927e-94d1-45e9-9521-f822260f7a1f 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 3e5f927e-94d1-45e9-9521-f822260f7a1f' 2024-03-30T11:00:16.563 INFO:tasks.workunit.client.0.smithi003.stdout:add osd7 3e5f927e-94d1-45e9-9521-f822260f7a1f 2024-03-30T11:00:16.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:00:16.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDA8Adm+xV/IhAA3cCSQYDBInB1Fd83P+owTQ== 2024-03-30T11:00:16.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDA8Adm+xV/IhAA3cCSQYDBInB1Fd83P+owTQ=="}' 2024-03-30T11:00:16.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e5f927e-94d1-45e9-9521-f822260f7a1f -i td/osd-recovery-scrub/7/new.json 2024-03-30T11:00:16.995 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:00:17.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-03-30T11:00:17.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDA8Adm+xV/IhAA3cCSQYDBInB1Fd83P+owTQ== --osd-uuid 3e5f927e-94d1-45e9-9521-f822260f7a1f 2024-03-30T11:00:17.040 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.041+0000 7f4b5f1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:17.080 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.081+0000 7f4b5f1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:17.081 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.082+0000 7f4b5f1a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:17.082 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.083+0000 7f4b5f1a8640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-30T11:00:17.082 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.083+0000 7f4b5f1a8640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-30T11:00:17.082 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.083+0000 7f4b5f1a8640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-30T11:00:17.155 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:17.156+0000 7f4b5f1a8640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-03-30T11:00:23.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-03-30T11:00:23.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:00:23.183 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd7 key to auth repository 2024-03-30T11:00:23.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-30T11:00:23.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:00:23.551 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.7 2024-03-30T11:00:23.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-30T11:00:23.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=9d2df085-a896-41cc-8cb8-078308a1d956 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.45029/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-30T11:00:23.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:00:23.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:00:23.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:00:23.581 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:23.582+0000 7fc2d1c37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:23.583 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:23.584+0000 7fc2d1c37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:23.584 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:23.585+0000 7fc2d1c37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:23.832 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:00:23.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:00:23.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:23.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:00:24.106 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:24.108+0000 7fc2d1c37640 -1 Falling back to public interface 2024-03-30T11:00:24.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:25.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:25.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:25.115 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:00:25.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:00:25.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:25.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:00:25.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:26.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:26.409 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:00:26.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:26.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:00:26.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:26.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:00:26.559 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:00:26.560+0000 7fc2d1c37640 -1 osd.7 0 log_to_monitors true 2024-03-30T11:00:26.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:00:27.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:00:27.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:00:27.779 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:00:27.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:00:27.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:00:27.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:00:28.087 INFO:tasks.workunit.client.0.smithi003.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/358525768,v1:127.0.0.1:6859/358525768] [v2:127.0.0.1:6860/358525768,v1:127.0.0.1:6861/358525768] exists,up 3e5f927e-94d1-45e9-9521-f822260f7a1f 2024-03-30T11:00:28.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:00:28.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:00:28.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:00:28.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-03-30T11:00:28.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-03-30T11:00:28.592 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'test' created 2024-03-30T11:00:28.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:00:29.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-03-30T11:00:29.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:00:29.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:00:29.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:00:29.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:00:29.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:00:29.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:00:29.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:00:29.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:00:29.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:00:29.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:00:29.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:00:29.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:00:29.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:00:29.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:00:29.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:00:30.091 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:00:30.092 INFO:tasks.workunit.client.0.smithi003.stderr:7' 2024-03-30T11:00:30.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:00:30.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:30.092 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:00:30.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-30T11:00:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-30T11:00:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-30T11:00:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:30.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:00:30.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-30T11:00:30.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-30T11:00:30.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976' 2024-03-30T11:00:30.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:30.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:00:30.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-30T11:00:30.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-30T11:00:30.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454' 2024-03-30T11:00:30.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:30.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:00:31.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-03-30T11:00:31.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-03-30T11:00:31.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931' 2024-03-30T11:00:31.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:31.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:00:31.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-03-30T11:00:31.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-03-30T11:00:31.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931 4-107374182409' 2024-03-30T11:00:31.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:31.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:00:31.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-30T11:00:31.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-30T11:00:31.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887' 2024-03-30T11:00:31.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:31.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:00:31.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-30T11:00:31.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-30T11:00:31.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887 6-150323855365' 2024-03-30T11:00:31.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:00:31.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:00:31.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-30T11:00:31.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-30T11:00:31.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931 4-107374182409 5-128849018887 6-150323855365 7-171798691842' 2024-03-30T11:00:31.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:31.782 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-30T11:00:31.782 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:31.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:00:31.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-30T11:00:31.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:31.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-30T11:00:31.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-30T11:00:31.784 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836499 2024-03-30T11:00:31.784 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:00:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-30T11:00:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:32.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-30T11:00:32.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:32.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:00:32.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-30T11:00:32.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:32.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-30T11:00:32.099 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672976 2024-03-30T11:00:32.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-30T11:00:32.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:00:32.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-30T11:00:32.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:32.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-30T11:00:32.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:32.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:00:32.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-30T11:00:32.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:32.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-30T11:00:32.440 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509454 2024-03-30T11:00:32.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-30T11:00:32.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:00:32.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-30T11:00:32.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:32.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-03-30T11:00:32.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:32.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:00:32.727 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-03-30T11:00:32.727 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:32.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-03-30T11:00:32.728 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345931 2024-03-30T11:00:32.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-03-30T11:00:32.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:00:33.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-03-30T11:00:33.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:33.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-03-30T11:00:33.018 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:33.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:00:33.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-03-30T11:00:33.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:33.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-03-30T11:00:33.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-03-30T11:00:33.020 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.4 seq 107374182409 2024-03-30T11:00:33.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:00:33.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-03-30T11:00:33.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:33.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-03-30T11:00:33.302 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:33.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:00:33.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-03-30T11:00:33.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:33.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-30T11:00:33.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-03-30T11:00:33.304 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.5 seq 128849018887 2024-03-30T11:00:33.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:00:33.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-03-30T11:00:33.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:33.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855365 2024-03-30T11:00:33.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:33.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:00:33.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855365 2024-03-30T11:00:33.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:33.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-30T11:00:33.682 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.6 seq 150323855365 2024-03-30T11:00:33.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-03-30T11:00:33.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:00:33.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-30T11:00:33.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:00:33.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-03-30T11:00:33.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:00:33.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:00:33.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-03-30T11:00:33.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:00:33.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-30T11:00:33.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-03-30T11:00:33.966 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.7 seq 171798691842 2024-03-30T11:00:33.966 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:00:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-03-30T11:00:34.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:00:34.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:00:34.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:00:34.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-03-30T11:00:34.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:00:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:00:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:00:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:00:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:00:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:00:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:00:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-03-30T11:00:34.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:00:34.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:00:34.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:00:35.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-03-30T11:00:35.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:00:35.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:00:35.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-03-30T11:00:35.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-03-30T11:00:35.249 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-03-30T11:00:35.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-03-30T11:00:35.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.45029 bs=1M count=50 2024-03-30T11:00:35.698 INFO:tasks.workunit.client.0.smithi003.stderr:50+0 records in 2024-03-30T11:00:35.698 INFO:tasks.workunit.client.0.smithi003.stderr:50+0 records out 2024-03-30T11:00:35.698 INFO:tasks.workunit.client.0.smithi003.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.138144 s, 380 MB/s 2024-03-30T11:00:35.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-03-30T11:00:35.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:35.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.45029 2024-03-30T11:00:37.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:37.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.45029 2024-03-30T11:00:38.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:38.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.45029 2024-03-30T11:00:40.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:40.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.45029 2024-03-30T11:00:41.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:41.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.45029 2024-03-30T11:00:43.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:43.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.45029 2024-03-30T11:00:44.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:44.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.45029 2024-03-30T11:00:46.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:46.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.45029 2024-03-30T11:00:47.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:47.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.45029 2024-03-30T11:00:48.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:48.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.45029 2024-03-30T11:00:50.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:50.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.45029 2024-03-30T11:00:51.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:51.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.45029 2024-03-30T11:00:53.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:53.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.45029 2024-03-30T11:00:54.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:54.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.45029 2024-03-30T11:00:56.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:56.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.45029 2024-03-30T11:00:57.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:57.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.45029 2024-03-30T11:00:59.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:00:59.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.45029 2024-03-30T11:01:00.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:00.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.45029 2024-03-30T11:01:02.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:02.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.45029 2024-03-30T11:01:03.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:03.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.45029 2024-03-30T11:01:05.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:05.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.45029 2024-03-30T11:01:06.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:06.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.45029 2024-03-30T11:01:08.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:08.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.45029 2024-03-30T11:01:09.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:09.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.45029 2024-03-30T11:01:11.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:11.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.45029 2024-03-30T11:01:12.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:12.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.45029 2024-03-30T11:01:14.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:14.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.45029 2024-03-30T11:01:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.45029 2024-03-30T11:01:17.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:17.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.45029 2024-03-30T11:01:18.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:18.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.45029 2024-03-30T11:01:20.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:20.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.45029 2024-03-30T11:01:21.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:21.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.45029 2024-03-30T11:01:23.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:23.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.45029 2024-03-30T11:01:24.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:24.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.45029 2024-03-30T11:01:26.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:26.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.45029 2024-03-30T11:01:27.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:27.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.45029 2024-03-30T11:01:29.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:29.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.45029 2024-03-30T11:01:30.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:30.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.45029 2024-03-30T11:01:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:32.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.45029 2024-03-30T11:01:33.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-30T11:01:33.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.45029 2024-03-30T11:01:35.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.45029 2024-03-30T11:01:35.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-03-30T11:01:35.638 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 3 2024-03-30T11:01:35.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-30T11:01:35.918 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:01:35.955 INFO:tasks.workunit.client.0.smithi003.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-30T11:00:30.929478+0000 51'26 51:58 [5] 5 [5] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.955 INFO:tasks.workunit.client.0.smithi003.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.372971+0000 51'13 51:46 [3] 3 [3] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.1d 2 0 0 0 0 81788928 0 0 20 0 20 active+clean 2024-03-30T11:00:30.372927+0000 51'20 51:41 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.372800+0000 51'13 51:37 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.232269+0000 51'13 51:37 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:30.232285+0000 0'0 51:24 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-30T11:00:30.928715+0000 51'26 51:49 [5] 5 [5] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.822958+0000 51'13 51:38 [6] 6 [6] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.956 INFO:tasks.workunit.client.0.smithi003.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:29.822954+0000 0'0 51:25 [6] 6 [6] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-30T11:00:30.928907+0000 51'39 51:64 [4] 4 [4] 4 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-30T11:00:30.372603+0000 51'52 51:79 [3] 3 [3] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-30T11:00:30.752210+0000 51'52 51:77 [4] 4 [4] 4 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.216130+0000 51'13 51:42 [0] 0 [0] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:30.232257+0000 0'0 51:25 [3] 3 [3] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.929183+0000 51'13 51:45 [5] 5 [5] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.378812+0000 51'13 51:35 [0] 0 [0] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.957 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.928738+0000 51'13 51:44 [7] 7 [7] 7 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.802512+0000 51'13 51:35 [7] 7 [7] 7 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.372603+0000 51'13 51:36 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.822849+0000 51'13 51:35 [6] 6 [6] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.929335+0000 51'13 51:35 [5] 5 [5] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.378595+0000 51'13 51:37 [0] 0 [0] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-30T11:00:29.216172+0000 51'26 51:49 [0] 0 [0] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.958 INFO:tasks.workunit.client.0.smithi003.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.928710+0000 51'13 51:34 [4] 4 [4] 4 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:30.752218+0000 0'0 51:23 [2] 2 [2] 2 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-30T11:00:30.373128+0000 51'39 51:61 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:29.913025+0000 0'0 51:21 [6] 6 [6] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.378400+0000 51'13 51:36 [0] 0 [0] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:30.752211+0000 0'0 51:21 [5] 5 [5] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:30.372793+0000 51'13 51:36 [3] 3 [3] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:00:30.373220+0000 0'0 51:22 [1] 1 [1] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-30T11:00:29.378720+0000 51'13 51:36 [0] 0 [0] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:35.959 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:01:35.960 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:01:35.960 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:01:35.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-03-30T11:01:36.041 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:01:36.041 INFO:tasks.workunit.client.0.smithi003.stdout: "osd_recovery_max_active": "0" 2024-03-30T11:01:36.041 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:01:36.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-03-30T11:01:36.304 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:01:36.304 INFO:tasks.workunit.client.0.smithi003.stdout: "osd_recovery_max_active_hdd": "3" 2024-03-30T11:01:36.304 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:01:36.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-03-30T11:01:36.410 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:01:36.410 INFO:tasks.workunit.client.0.smithi003.stdout: "osd_recovery_max_active_ssd": "10" 2024-03-30T11:01:36.410 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:01:36.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-03-30T11:01:36.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-30T11:01:36.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-30T11:01:36.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-30T11:01:36.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-30T11:01:36.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-30T11:01:36.772 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:01:36.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-30T11:01:36.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-30T11:01:38.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-03-30T11:01:38.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-03-30T11:01:38.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-03-30T11:01:38.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-30T11:01:38.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-30T11:01:38.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-30T11:01:38.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-30T11:01:38.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-30T11:01:39.068 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:01:39.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-30T11:01:39.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-30T11:01:41.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-03-30T11:01:41.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-03-30T11:01:41.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-03-30T11:01:41.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-30T11:01:41.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-30T11:01:41.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-30T11:01:41.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-30T11:01:41.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-30T11:01:41.363 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:01:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-03-30T11:01:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-03-30T11:01:41.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-30T11:01:41.641 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:01:41.641 INFO:tasks.workunit.client.0.smithi003.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-30T11:01:37.448404+0000 51'26 54:68 [5,0,2] 5 [5,0,2] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.641 INFO:tasks.workunit.client.0.smithi003.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-03-30T11:01:37.229818+0000 51'13 54:51 [3,0,7] 3 [3,0,7] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-30T11:01:37.597511+0000 51'26 54:57 [1,3,0] 1 [1,3,0] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.821638+0000 51'13 54:46 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.649740+0000 51'13 54:46 [1,6,2] 1 [1,6,2] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:01:37.775771+0000 0'0 54:35 [1,4,5] 1 [1,4,5] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-30T11:01:38.321694+0000 51'26 54:58 [5,4,3] 5 [5,4,3] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.596119+0000 51'13 54:47 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.821457+0000 51'13 54:48 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-30T11:01:37.356201+0000 51'39 54:74 [4,2,6] 4 [4,2,6] 4 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 activating 2024-03-30T11:01:36.905383+0000 51'52 54:83 [3,0,6] 3 [3,0,6] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-30T11:01:37.968557+0000 51'52 54:88 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.822279+0000 51'13 54:52 [0,7,1] 0 [0,7,1] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 activating 2024-03-30T11:01:36.905207+0000 0'0 54:30 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.642 INFO:tasks.workunit.client.0.smithi003.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.774729+0000 51'13 54:55 [5,0,4] 5 [5,0,4] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-03-30T11:01:37.398014+0000 51'13 54:41 [0,3,4] 0 [0,3,4] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.821616+0000 51'13 54:54 [7,0,2] 7 [7,0,2] 7 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.969374+0000 51'13 54:44 [7,4,2] 7 [7,4,2] 7 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.775563+0000 51'13 54:45 [1,4,2] 1 [1,4,2] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:38.044586+0000 51'13 54:44 [6,0,3] 6 [6,0,3] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.550039+0000 51'13 54:44 [5,7,6] 5 [5,7,6] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-30T11:01:37.597291+0000 51'13 54:46 [0,1,3] 0 [0,1,3] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-30T11:01:37.596446+0000 51'26 54:58 [0,1,6] 0 [0,1,6] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-30T11:01:36.860027+0000 51'13 54:45 [4,6,3] 4 [4,6,3] 4 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:01:37.596469+0000 0'0 54:33 [2,0,4] 2 [2,0,4] 2 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-30T11:01:37.596447+0000 51'39 54:70 [1,5,3] 1 [1,5,3] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:01:37.398084+0000 0'0 54:32 [6,0,7] 6 [6,0,7] 6 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 activating+degraded 2024-03-30T11:01:37.447443+0000 51'13 54:41 [0,4,5] 0 [0,4,5] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.643 INFO:tasks.workunit.client.0.smithi003.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:01:37.262798+0000 0'0 54:32 [5,2,3] 5 [5,2,3] 5 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:41.644 INFO:tasks.workunit.client.0.smithi003.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 activating 2024-03-30T11:01:36.905851+0000 51'13 54:40 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.644 INFO:tasks.workunit.client.0.smithi003.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-30T11:01:37.774972+0000 0'0 54:33 [1,7,4] 1 [1,7,4] 1 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 periodic scrub scheduled @ 2024-03-31T11:00:28.418834+0000 0 0 2024-03-30T11:01:41.644 INFO:tasks.workunit.client.0.smithi003.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-30T11:01:37.449041+0000 51'13 54:46 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-30T11:00:28.418834+0000 0'0 2024-03-30T11:00:28.418834+0000 0 0 no scrub is scheduled 0 0 2024-03-30T11:01:41.644 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:01:41.644 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:01:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:01:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-03-30T11:01:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-03-30T11:01:41.652 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-03-30T11:01:41.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-03-30T11:01:41.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66447"' 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66447' 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-03-30T11:01:41.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66450: /' 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66452"' 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66452' 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-03-30T11:01:41.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66457: /' 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66459"' 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66459' 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66464: /' 2024-03-30T11:01:41.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66466"' 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66466' 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-03-30T11:01:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66470: /' 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66473"' 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66473' 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66478: /' 2024-03-30T11:01:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-03-30T11:01:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66479"' 2024-03-30T11:01:41.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66479' 2024-03-30T11:01:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-03-30T11:01:41.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-03-30T11:01:41.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66484"' 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66484' 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66483: /' 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66494"' 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66494' 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66501"' 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66501' 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-03-30T11:01:41.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-03-30T11:01:41.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66503"' 2024-03-30T11:01:41.673 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66503' 2024-03-30T11:01:41.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-03-30T11:01:41.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-03-30T11:01:41.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66510"' 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66510' 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66519"' 2024-03-30T11:01:41.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66490: /' 2024-03-30T11:01:41.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66519' 2024-03-30T11:01:41.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66497: /' 2024-03-30T11:01:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66506: /' 2024-03-30T11:01:41.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66508: /' 2024-03-30T11:01:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66527: /' 2024-03-30T11:01:41.700 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-03-30T11:01:41.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-03-30T11:01:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-03-30T11:01:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66529"' 2024-03-30T11:01:41.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66529' 2024-03-30T11:01:41.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66515: /' 2024-03-30T11:01:41.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-03-30T11:01:41.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-03-30T11:01:41.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66533"' 2024-03-30T11:01:41.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66533' 2024-03-30T11:01:41.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-03-30T11:01:41.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-03-30T11:01:41.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66535"' 2024-03-30T11:01:41.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66535' 2024-03-30T11:01:41.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66541"' 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66541' 2024-03-30T11:01:41.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66538: /' 2024-03-30T11:01:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66540: /' 2024-03-30T11:01:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-03-30T11:01:41.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66547"' 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66547' 2024-03-30T11:01:41.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-03-30T11:01:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66544: /' 2024-03-30T11:01:41.747 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-03-30T11:01:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66550: /' 2024-03-30T11:01:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-03-30T11:01:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66553"' 2024-03-30T11:01:41.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66553' 2024-03-30T11:01:41.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-03-30T11:01:41.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-03-30T11:01:41.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66562"' 2024-03-30T11:01:41.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66562' 2024-03-30T11:01:41.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-03-30T11:01:41.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66566: /' 2024-03-30T11:01:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-03-30T11:01:41.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66552: /' 2024-03-30T11:01:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-03-30T11:01:41.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-03-30T11:01:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-03-30T11:01:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66569"' 2024-03-30T11:01:41.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66569' 2024-03-30T11:01:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-03-30T11:01:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-03-30T11:01:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66577"' 2024-03-30T11:01:41.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66577' 2024-03-30T11:01:41.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-03-30T11:01:41.785 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-03-30T11:01:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-03-30T11:01:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66584"' 2024-03-30T11:01:41.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66584' 2024-03-30T11:01:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66574: /' 2024-03-30T11:01:41.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-03-30T11:01:41.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-03-30T11:01:41.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66580: /' 2024-03-30T11:01:41.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66587: /' 2024-03-30T11:01:41.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-03-30T11:01:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-03-30T11:01:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66598"' 2024-03-30T11:01:41.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66598' 2024-03-30T11:01:41.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-03-30T11:01:41.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66590: /' 2024-03-30T11:01:41.825 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-03-30T11:01:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-03-30T11:01:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66608"' 2024-03-30T11:01:41.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66608' 2024-03-30T11:01:41.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66601: /' 2024-03-30T11:01:41.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-03-30T11:01:41.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-03-30T11:01:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-03-30T11:01:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66616"' 2024-03-30T11:01:41.834 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66616' 2024-03-30T11:01:41.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-03-30T11:01:41.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-03-30T11:01:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-03-30T11:01:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66619"' 2024-03-30T11:01:41.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66619' 2024-03-30T11:01:41.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66624: /' 2024-03-30T11:01:41.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66612: /' 2024-03-30T11:01:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-03-30T11:01:41.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-03-30T11:01:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-03-30T11:01:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66631"' 2024-03-30T11:01:41.862 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66631' 2024-03-30T11:01:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-03-30T11:01:41.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-03-30T11:01:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-03-30T11:01:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66636"' 2024-03-30T11:01:41.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66636' 2024-03-30T11:01:41.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66622: /' 2024-03-30T11:01:41.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66639: /' 2024-03-30T11:01:41.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66643"' 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66643' 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-03-30T11:01:41.892 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-03-30T11:01:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-03-30T11:01:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66652"' 2024-03-30T11:01:41.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66652' 2024-03-30T11:01:41.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-03-30T11:01:41.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66648: /' 2024-03-30T11:01:41.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-03-30T11:01:41.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66655: /' 2024-03-30T11:01:41.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-03-30T11:01:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66667"' 2024-03-30T11:01:41.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66667' 2024-03-30T11:01:41.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-30T11:01:41.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-03-30T11:01:41.909 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-03-30T11:01:41.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66672: /' 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66678"' 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66678' 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 66447 66452 66459 66466 66473 66479 66484 66494 66501 66503 66510 66519 66529 66533 66535 66541 66547 66553 66562 66569 66577 66584 66598 66608 66616 66619 66631 66636 66643 66652 66667 66678' 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:41.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66447 2024-03-30T11:01:41.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-03-30T11:01:41.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66688: /' 2024-03-30T11:01:41.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-03-30T11:01:41.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-03-30T11:01:41.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66692: /' 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:47.764 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: instructing pg 1.1b on osd.0 to scrub 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: dumped pgs 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:47.766 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.169 INFO:tasks.workunit.client.0.smithi003.stderr:66648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-30T11:01:49.169 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.169 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.169 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-30T11:01:49.170 INFO:tasks.workunit.client.0.smithi003.stderr:66648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-30T11:01:49.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: instructing pg 1.10 on osd.5 to scrub 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: dumped pgs 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:49.540 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:66552: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:49.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:49.598 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: instructing pg 1.3 on osd.4 to scrub 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: dumped pgs 2024-03-30T11:01:49.599 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:66470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:49.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:49.739 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.740 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: instructing pg 1.0 on osd.7 to scrub 2024-03-30T11:01:49.741 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:49.742 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:49.742 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-03-30T11:01:49.742 INFO:tasks.workunit.client.0.smithi003.stderr:66450: dumped pgs 2024-03-30T11:01:49.742 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.742 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-03-30T11:01:49.742 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:01:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:01:49.744 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.744 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:01:49.744 INFO:tasks.workunit.client.0.smithi003.stderr:66450: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.744 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.744 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.744 INFO:tasks.workunit.client.0.smithi003.stderr:66450: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:49.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:49.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:49.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:49.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:49.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:49.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66452 2024-03-30T11:01:49.848 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-03-30T11:01:49.848 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-03-30T11:01:49.848 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-30T11:01:49.848 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.848 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-30T11:01:49.849 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.849 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.849 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:49.849 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-03-30T11:01:49.849 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:49.849 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-03-30T11:01:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:66574: instructing pg 1.12 on osd.0 to scrub 2024-03-30T11:01:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-03-30T11:01:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:66574: dumped pgs 2024-03-30T11:01:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-03-30T11:01:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:49.852 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:49.852 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:49.852 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:49.852 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:49.852 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-03-30T11:01:49.852 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-30T11:01:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-30T11:01:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:66574: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:66574: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:49.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:49.881 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-03-30T11:01:49.881 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-03-30T11:01:49.881 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-30T11:01:49.881 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-30T11:01:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-03-30T11:01:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-03-30T11:01:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:66506: instructing pg 1.8 on osd.5 to scrub 2024-03-30T11:01:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-03-30T11:01:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:66506: dumped pgs 2024-03-30T11:01:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-03-30T11:01:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.885 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:49.885 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:49.885 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:49.885 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:49.885 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:49.885 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-03-30T11:01:49.886 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-30T11:01:49.886 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.886 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-30T11:01:49.886 INFO:tasks.workunit.client.0.smithi003.stderr:66506: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.886 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.886 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:66506: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:49.977 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-03-30T11:01:49.977 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: instructing pg 1.2 on osd.0 to scrub 2024-03-30T11:01:49.978 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: dumped pgs 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:49.979 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:66464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:49.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: instructing pg 1.1d on osd.1 to scrub 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: dumped pgs 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-03-30T11:01:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-30T11:01:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-30T11:01:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.251 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.251 INFO:tasks.workunit.client.0.smithi003.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.329 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-03-30T11:01:50.330 INFO:tasks.workunit.client.0.smithi003.stderr:66580: instructing pg 1.13 on osd.4 to scrub 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: dumped pgs 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.331 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-03-30T11:01:50.332 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-30T11:01:50.332 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.332 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-30T11:01:50.346 INFO:tasks.workunit.client.0.smithi003.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: g66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-03-30T11:01:50.346 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: instructing pg 1.1f on osd.5 to scrub 2024-03-30T11:01:50.347 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: dumped pgs 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.348 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:66688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: instructing pg 1.4 on osd.3 to scrub 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: dumped pgs 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.375 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:66478: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.516 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: instructing pg 1.e on osd.1 to scrub 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: dumped pgs 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-03-30T11:01:50.517 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:66544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:66544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.541 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-03-30T11:01:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:66508: instructing pg 1.9 on osd.1 to scrub 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: dumped pgs 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-30T11:01:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-30T11:01:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:66508: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:01:46.481187+0000 '>' '' 2024-03-30T11:01:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:66508: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: instructing pg 1.7 on osd.6 to scrub 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: dumped pgs 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.601 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:66497: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-03-30T11:01:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-03-30T11:01:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-30T11:01:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: instructing pg 1.b on osd.1 to scrub 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: dumped pgs 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.650 INFO:tasks.workunit.client.0.smithi003.stderr:66527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.666 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-03-30T11:01:50.666 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-03-30T11:01:50.666 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-30T11:01:50.666 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.666 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: instructing pg 1.19 on osd.3 to scrub 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.667 INFO:tasks.workunit.client.0.smithi003.stderr:66624: dumped pgs 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-03-30T11:01:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-30T11:01:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:66624: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-30T11:01:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.670 INFO:tasks.workunit.client.0.smithi003.stderr:66624: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.688 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-03-30T11:01:50.689 INFO:tasks.workunit.client.0.smithi003.stderr:66622: instructing pg 1.18 on osd.5 to scrub 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: dumped pgs 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-30T11:01:50.690 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.691 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.691 INFO:tasks.workunit.client.0.smithi003.stderr:66622: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-30T11:01:50.691 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.691 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.691 INFO:tasks.workunit.client.0.smithi003.stderr:66622: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.700 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: instructing pg 1.1e on osd.3 to scrub 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: dumped pgs 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.701 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:66692: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.766 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: instructing pg 1.a on osd.1 to scrub 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: dumped pgs 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.767 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:66515: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.778 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-03-30T11:01:50.778 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-03-30T11:01:50.778 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-30T11:01:50.778 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.778 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: instructing pg 1.c on osd.3 to scrub 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: dumped pgs 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.780 INFO:tasks.workunit.client.0.smithi003.stderr:66540: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.802 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: instructing pg 1.16 on osd.6 to scrub 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: dumped pgs 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.803 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:66601: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-03-30T11:01:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: instructing pg 1.f on osd.6 to scrub 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: dumped pgs 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:66550: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.873 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: instructing pg 1.6 on osd.6 to scrub 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: dumped pgs 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:66490: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:50.907 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: instructing pg 1.15 on osd.1 to scrub 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: dumped pgs 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:50.908 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:66590: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:50.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: instructing pg 1.1c on osd.0 to scrub 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: dumped pgs 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.302 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-03-30T11:01:51.308 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: instructing pg 1.1 on osd.5 to scrub 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: dumped pgs 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.309 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-03-30T11:01:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-30T11:01:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:66457: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-30T11:01:51.312 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scr66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-03-30T11:01:51.312 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-03-30T11:01:51.312 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-30T11:01:51.312 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.312 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: instructing pg 1.11 on osd.0 to scrub 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: dumped pgs 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-03-30T11:01:51.313 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:66566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:ub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:66457: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66459 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66466 2024-03-30T11:01:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66473 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66479 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: instructing pg 1.1a on osd.1 to scrub 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: dumped pgs 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:01:47.438817+0000 '>' '' 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:66639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.343 INFO:tasks.workunit.client.0.smithi003.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.343 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.344 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.344 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-30T11:01:51.344 INFO:tasks.workunit.client.0.smithi003.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-30T11:01:51.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-30T11:01:51.415 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-03-30T11:01:51.415 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-03-30T11:01:51.415 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-30T11:01:51.415 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.415 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: instructing pg 1.5 on osd.4 to scrub 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: dumped pgs 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.416 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:66483: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66484 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66494 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66501 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66503 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66510 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66519 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66529 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66533 2024-03-30T11:01:51.420 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: instructing pg 1.17 on osd.0 to scrub 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.421 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: dumped pgs 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:66612: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-30T11:01:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:66612: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-03-30T11:01:51.475 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: instructing pg 1.14 on osd.2 to scrub 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: dumped pgs 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.476 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.538 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-03-30T11:01:51.538 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-03-30T11:01:51.538 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: instructing pg 1.d on osd.7 to scrub 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-30T11:01:51.539 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: dumped pgs 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-30T11:00:28.418834+0000 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-30T11:01:51.540 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-30T11:00:28.418834+0000 '>' '' 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:66538: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66535 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66541 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66547 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66553 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66562 2024-03-30T11:01:51.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66569 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66577 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66584 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66598 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66608 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66616 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66619 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66631 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66636 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66643 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66652 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66667 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66678 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-03-30T11:01:51.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.5.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.6.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.1.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.0.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.4.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.3.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.7.pid 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr:td/osd-recovery-scrub/osd.2.pid' 2024-03-30T11:01:51.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53590 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:61516 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:62893 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:56012 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:53674 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:54631 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:60140 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:58764 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:64269 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:57388' 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53590 61516 62893 56012 53674 54631 60140 58764 64269 57388 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:01:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:01:51.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:01:51.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:01:56.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:01:56.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-03-30T11:01:56.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-03-30T11:01:56.851 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-03-30T11:01:56.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-03-30T11:01:56.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-03-30T11:01:56.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-03-30T11:01:56.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-03-30T11:01:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-03-30T11:01:56.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-03-30T11:01:56.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-03-30T11:01:56.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-03-30T11:01:56.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-03-30T11:01:56.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-03-30T11:01:56.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-03-30T11:01:56.884 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-03-30T11:01:56.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-03-30T11:01:56.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-03-30T11:01:56.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-03-30T11:01:56.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-03-30T11:01:56.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-03-30T11:01:56.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-03-30T11:01:56.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-03-30T11:01:56.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-03-30T11:01:56.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T11:01:56.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-03-30T11:01:56.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-03-30T11:01:56.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-03-30T11:01:56.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:01:56.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:01:56.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:01:56.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:01:56.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:01:56.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:01:56.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:01:56.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:01:56.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:01:56.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:56.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:01:56.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:01:56.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:56.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:01:56.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:01:56.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-30T11:01:56.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:01:56.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:56.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T11:01:56.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45029 2024-03-30T11:01:56.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:01:56.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:01:56.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-03-30T11:01:56.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-03-30T11:01:56.951 INFO:tasks.workunit.client.0.smithi003.stdout:TEST PASSED 2024-03-30T11:01:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-03-30T11:01:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-30T11:01:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-03-30T11:01:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-30T11:01:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-30T11:01:56.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:01:56.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:01:56.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:01:56.954 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:01:56.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:01:56.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:01:56.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:01:56.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:56.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:01:56.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:01:56.956 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:56.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:01:56.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-30T11:01:56.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-30T11:01:56.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:01:56.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:56.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.45029 2024-03-30T11:01:56.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.45029 2024-03-30T11:01:56.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:01:56.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:01:56.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-30T11:01:56.961 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T11:01:56.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T11:01:57.030 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-03-30T11:01:57.031 DEBUG:teuthology.orchestra.run.smithi003:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-03-30T11:01:57.092 INFO:tasks.workunit.client.0.smithi003.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-30T11:01:57.096 INFO:tasks.workunit.client.0.smithi003.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-03-30T11:01:57.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-03-30T11:01:57.098 INFO:tasks.workunit.client.0.smithi003.stdout:This test is disabled 2024-03-30T11:01:57.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-03-30T11:01:57.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-30T11:01:57.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-03-30T11:01:57.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-03-30T11:01:57.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-30T11:01:57.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:01:57.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:01:57.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:01:57.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:01:57.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:01:57.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:57.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:01:57.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:01:57.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:57.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:01:57.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-30T11:01:57.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-03-30T11:01:57.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:01:57.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:57.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69918 2024-03-30T11:01:57.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69918 2024-03-30T11:01:57.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:01:57.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:01:57.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-30T11:01:57.106 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-30T11:01:57.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T11:01:57.172 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-03-30T11:01:57.173 DEBUG:teuthology.orchestra.run.smithi003:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-03-30T11:01:57.232 INFO:tasks.workunit.client.0.smithi003.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-30T11:01:57.232 INFO:tasks.workunit.client.0.smithi003.stderr:++ TIMEOUT=300 2024-03-30T11:01:57.232 INFO:tasks.workunit.client.0.smithi003.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-03-30T11:01:57.232 INFO:tasks.workunit.client.0.smithi003.stderr:++ MAX_TIMEOUT=15 2024-03-30T11:01:57.232 INFO:tasks.workunit.client.0.smithi003.stderr:++ PG_NUM=4 2024-03-30T11:01:57.232 INFO:tasks.workunit.client.0.smithi003.stderr:++ TMPDIR=/tmp 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ type xmlstarlet 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ XMLSTARLET=xmlstarlet 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:+++ uname 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' Linux = FreeBSD ']' 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ SED=sed 2024-03-30T11:01:57.233 INFO:tasks.workunit.client.0.smithi003.stderr:++ AWK=awk 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:+++ stty -a 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:+++ head -1 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:++ termwidth= 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-03-30T11:01:57.234 INFO:tasks.workunit.client.0.smithi003.stderr:++ DIFFCOLOPTS='-y ' 2024-03-30T11:01:57.235 INFO:tasks.workunit.client.0.smithi003.stderr:++ KERNCORE=kernel.core_pattern 2024-03-30T11:01:57.235 INFO:tasks.workunit.client.0.smithi003.stderr:++ EXTRA_OPTS= 2024-03-30T11:01:57.237 INFO:tasks.workunit.client.0.smithi003.stderr:++ test '' = TESTS 2024-03-30T11:01:57.237 INFO:tasks.workunit.client.0.smithi003.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-03-30T11:01:57.237 INFO:tasks.workunit.client.0.smithi003.stderr:++ uname 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' Linux = FreeBSD ']' 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr:+ use_ec_overwrite=true 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr:+ getjson=no 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr:+ jqfilter='def walk(f): 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:01:57.239 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:01:57.240 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:01:57.240 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:01:57.240 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:01:57.240 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T11:01:57.240 INFO:tasks.workunit.client.0.smithi003.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:01:57.244 INFO:tasks.workunit.client.0.smithi003.stderr:+ main osd-scrub-repair 2024-03-30T11:01:57.244 INFO:tasks.workunit.client.0.smithi003.stderr:+ local dir=td/osd-scrub-repair 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:+ shift 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:+ shopt -s -o xtrace 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-03-30T11:01:57.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none ' 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-03-30T11:01:57.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_bluestore_basic 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_bluestore_failed 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_bluestore_scrub 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_bluestore_tag 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_erasure_coded_appends 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_and_repair_lrc_appends 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-03-30T11:01:57.249 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_and_repair_replicated 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_scrub_erasure_appends 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_scrub_replicated 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_corrupt_snapset_scrub_rep 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_list_missing_erasure_coded_appends 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_list_missing_erasure_coded_overwrites 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_periodic_scrub_replicated 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_repair_stats 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_repair_stats_ec 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_request_scrub_priority 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_scrub_warning 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_skip_non_repair_during_recovery 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_unfound_erasure_coded_appends 2024-03-30T11:01:57.250 INFO:tasks.workunit.client.0.smithi003.stderr:TEST_unfound_erasure_coded_overwrites' 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:01:57.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:01:57.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:01:57.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:01:57.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:01:57.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:01:57.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:01:57.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:01:57.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:01:57.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:01:57.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:01:57.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:57.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:57.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:01:57.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:01:57.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:01:57.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:01:57.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:01:57.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:57.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-03-30T11:01:57.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-30T11:01:57.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:01:57.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:01:57.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:01:57.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:01:57.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:01:57.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-30T11:01:57.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:01:57.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:01:57.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:01:57.635 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:01:57.635 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:57.636 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:57.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:01:57.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-30T11:01:57.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:57.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:01:57.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:01:57.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:01:57.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:01:57.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:01:57.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:01:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:01:58.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:01:58.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:01:58.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:01:58.797 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:01:58.797 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:58.797 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:58.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:01:58.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:01:58.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:01:58.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-30T11:01:58.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:01:58.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:01:58.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:01:58.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:01:58.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:01:58.829 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:01:58.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:01:58.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9987ee24-fdc7-4021-9292-1bee932088aa 2024-03-30T11:01:58.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9987ee24-fdc7-4021-9292-1bee932088aa' 2024-03-30T11:01:58.831 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 9987ee24-fdc7-4021-9292-1bee932088aa 2024-03-30T11:01:58.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:01:58.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAm8QdmfoqOMhAAYU1NHELcJbS/9870bcw3YA== 2024-03-30T11:01:58.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAm8QdmfoqOMhAAYU1NHELcJbS/9870bcw3YA=="}' 2024-03-30T11:01:58.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9987ee24-fdc7-4021-9292-1bee932088aa -i td/osd-scrub-repair/0/new.json 2024-03-30T11:01:59.077 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:01:59.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:01:59.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAm8QdmfoqOMhAAYU1NHELcJbS/9870bcw3YA== --osd-uuid 9987ee24-fdc7-4021-9292-1bee932088aa 2024-03-30T11:01:59.118 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.120+0000 7f913f410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:01:59.127 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.130+0000 7f913f410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:01:59.129 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.131+0000 7f913f410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:01:59.129 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.131+0000 7f913f410640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:01:59.129 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.131+0000 7f913f410640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:01:59.129 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.131+0000 7f913f410640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:01:59.219 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:01:59.221+0000 7f913f410640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:02:04.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:02:04.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:02:04.808 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:02:04.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:02:04.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:02:05.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:02:05.184 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:02:05.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-30T11:02:05.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:02:05.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:02:05.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:02:05.212 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:05.214+0000 7f8b05519640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:05.214 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:05.216+0000 7f8b05519640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:05.215 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:05.217+0000 7f8b05519640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:05.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:02:05.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:02:05.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:05.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:05.729 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:05.732+0000 7f8b05519640 -1 Falling back to public interface 2024-03-30T11:02:05.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:06.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:06.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:06.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:02:06.754 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:02:06.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:06.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:07.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:08.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:08.052 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:02:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:02:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:08.066 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:08.068+0000 7f8b05519640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:02:08.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:09.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:09.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:09.493 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:02:09.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:02:09.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:09.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:09.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:10.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:10.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:10.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:02:10.798 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:02:10.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:10.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:11.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:12.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:12.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:12.093 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:02:12.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:02:12.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:12.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:12.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:13.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:13.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:13.386 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:02:13.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:02:13.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:13.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:13.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:14.757 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:02:14.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:02:14.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:14.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:15.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:16.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:16.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:16.050 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:02:16.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:02:16.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:16.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:16.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:17.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:17.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:17.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:02:17.344 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:02:17.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:17.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:17.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:18.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:18.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:18.638 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:02:18.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:02:18.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:18.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:19.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:20.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:20.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:20.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:02:20.237 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:02:20.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:20.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:21.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:22.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:22.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:22.033 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:02:22.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:02:22.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:22.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:22.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:23.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:23.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:23.407 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:02:23.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:02:23.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:23.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:24.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:25.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:25.026 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:02:25.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:25.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:02:25.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:25.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:25.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:26.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:26.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:26.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:02:26.628 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:02:26.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:26.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:26.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:27.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:27.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:27.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:02:27.946 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:02:27.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:27.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:28.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:29.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:29.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:29.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:02:29.314 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:02:29.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:29.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:02:29.605 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/374173642,v1:127.0.0.1:6803/374173642] [v2:127.0.0.1:6804/374173642,v1:127.0.0.1:6805/374173642] exists,up 9987ee24-fdc7-4021-9292-1bee932088aa 2024-03-30T11:02:29.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:02:29.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:02:29.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f44bd1f-61b4-48c7-812e-08da329809b1 2024-03-30T11:02:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5f44bd1f-61b4-48c7-812e-08da329809b1' 2024-03-30T11:02:29.612 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 5f44bd1f-61b4-48c7-812e-08da329809b1 2024-03-30T11:02:29.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:02:29.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBF8QdmzhhiJRAAZNkbKn4+kgEYLsOnJ4ntyw== 2024-03-30T11:02:29.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBF8QdmzhhiJRAAZNkbKn4+kgEYLsOnJ4ntyw=="}' 2024-03-30T11:02:29.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f44bd1f-61b4-48c7-812e-08da329809b1 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:02:29.991 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:02:30.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:02:30.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBF8QdmzhhiJRAAZNkbKn4+kgEYLsOnJ4ntyw== --osd-uuid 5f44bd1f-61b4-48c7-812e-08da329809b1 2024-03-30T11:02:30.032 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.034+0000 7f54d360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:30.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.061+0000 7f54d360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:30.059 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.062+0000 7f54d360d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.062+0000 7f54d360d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:02:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.062+0000 7f54d360d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:02:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.062+0000 7f54d360d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:02:30.134 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:30.136+0000 7f54d360d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:02:36.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:02:36.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:02:36.099 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:02:36.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:02:36.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:02:36.466 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:02:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:02:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-30T11:02:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:02:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:02:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:02:36.494 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:36.497+0000 7f534e411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:36.496 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:36.499+0000 7f534e411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:36.497 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:36.500+0000 7f534e411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:36.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:02:36.752 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:02:36.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:36.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:37.011 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:37.014+0000 7f534e411640 -1 Falling back to public interface 2024-03-30T11:02:37.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:38.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:38.030 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:02:38.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:38.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:02:38.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:38.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:38.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:39.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:39.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:39.327 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:02:39.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:02:39.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:39.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:39.332 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:02:39.335+0000 7f534e411640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:02:39.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:40.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:40.633 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:02:40.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:40.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:02:40.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:40.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:40.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:41.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:41.955 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:02:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:02:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:41.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:42.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:43.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:43.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:43.328 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:02:43.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:02:43.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:43.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:43.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:44.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:44.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:44.619 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:02:44.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:02:44.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:44.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:44.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:45.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:45.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:45.938 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:02:45.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:02:45.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:45.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:47.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:47.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:47.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:02:47.314 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:02:47.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:47.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:47.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:48.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:48.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:48.605 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:02:48.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:02:48.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:48.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:49.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:49.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:49.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:02:49.936 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:02:49.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:49.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:50.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:51.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:51.339 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:02:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:02:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:51.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:52.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:53.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:53.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:53.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:02:53.260 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:02:53.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:53.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:54.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:55.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:55.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:55.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:02:55.005 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:02:55.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:55.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:55.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:56.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:56.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:56.421 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:02:56.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:02:56.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:56.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:57.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:58.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:58.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:58.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:02:58.035 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:02:58.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:58.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:02:58.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:02:59.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:02:59.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:02:59.687 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:02:59.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:02:59.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:02:59.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:03:00.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:03:01.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:03:01.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:01.076 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:03:01.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:03:01.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:01.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:03:01.371 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/389348829,v1:127.0.0.1:6811/389348829] [v2:127.0.0.1:6812/389348829,v1:127.0.0.1:6813/389348829] exists,up 5f44bd1f-61b4-48c7-812e-08da329809b1 2024-03-30T11:03:01.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:03:01.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:03:01.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:03:01.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-03-30T11:03:01.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:03:01.717 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:03:01.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:03:01.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:03:02.243 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:03:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:03:03.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:03:04.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-03-30T11:03:04.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:03:04.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:03:04.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:03:04.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:03:04.669 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:03:04.669 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:03:04.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:03:04.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:03:04.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:03:04.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:03:04.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:03:04.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:03:04.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:03:04.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:03:04.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:03:05.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:03:05.044 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:03:05.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:03:05.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:03:05.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:03:05.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-30T11:03:05.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-30T11:03:05.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-30T11:03:05.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:03:05.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:03:05.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-30T11:03:05.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-30T11:03:05.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672966' 2024-03-30T11:03:05.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:03:05.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-30T11:03:05.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:03:05.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:03:05.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-30T11:03:05.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:03:05.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-30T11:03:05.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-30T11:03:05.295 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836493 2024-03-30T11:03:05.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:05.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-30T11:03:05.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:03:06.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:03:06.577 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:06.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-30T11:03:06.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:03:07.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:03:07.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:08.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-30T11:03:08.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:03:08.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-30T11:03:08.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:03:08.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:03:08.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-30T11:03:08.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:03:08.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-30T11:03:08.186 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672966 2024-03-30T11:03:08.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-30T11:03:08.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:03:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-03-30T11:03:08.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:03:08.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:03:08.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:03:08.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:03:08.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:03:08.862 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:03:08.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:03:08.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:03:08.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:03:08.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:03:08.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:03:09.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:03:09.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:03:09.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:03:09.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:03:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:03:09.881 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:03:09.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:03:10.383 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:03:10.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:03:10.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:03:10.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-30T11:03:10.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:03:10.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:03:10.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-30T11:03:10.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:03:11.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:03:11.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:03:11.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-30T11:03:12.778 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-30T11:03:13.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:03:13.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:03:13.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:03:13.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:03:13.298 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:03:13.299 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:03:13.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:03:13.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:03:13.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:03:13.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:03:13.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:03:13.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:03:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:13.329+0000 7f8a15f1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:13.329 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:13.332+0000 7f8a15f1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:13.330 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:13.333+0000 7f8a15f1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:13.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:03:13.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:03:13.847 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:13.850+0000 7f8a15f1b640 -1 Falling back to public interface 2024-03-30T11:03:13.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:03:14.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:03:14.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:14.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:03:14.876 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:03:14.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:14.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:03:15.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:03:16.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:03:16.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:16.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:03:16.170 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:03:16.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:16.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:03:16.202 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:16.205+0000 7f8a15f1b640 -1 osd.0 22 log_to_monitors true 2024-03-30T11:03:16.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:03:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:03:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:03:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:03:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/3127445571,v1:127.0.0.1:6803/3127445571] [v2:127.0.0.1:6804/3127445571,v1:127.0.0.1:6805/3127445571] exists,up 9987ee24-fdc7-4021-9292-1bee932088aa 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:03:17.755 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:03:17.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:03:17.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:03:17.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:03:17.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:03:17.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:03:17.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:03:17.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:03:17.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:03:17.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:03:18.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:03:18.116 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:03:18.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:03:18.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:03:18.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:03:18.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-30T11:03:18.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-30T11:03:18.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-30T11:03:18.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:03:18.263 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:03:18.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-30T11:03:18.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-30T11:03:18.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-03-30T11:03:18.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:03:18.369 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-30T11:03:18.369 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:03:18.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:03:18.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-30T11:03:18.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:03:18.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-30T11:03:18.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-30T11:03:18.371 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 111669149698 2024-03-30T11:03:18.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:18.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-30T11:03:18.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:03:19.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:03:19.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:19.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-30T11:03:19.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:03:19.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-30T11:03:19.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:03:19.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:03:19.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-30T11:03:19.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:03:19.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-30T11:03:19.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-30T11:03:19.947 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672970 2024-03-30T11:03:19.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:03:20.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-30T11:03:20.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:03:20.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:03:20.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:03:20.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:03:20.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:03:20.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:03:20.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:03:20.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:03:20.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:03:20.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:03:20.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:03:20.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:03:20.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:03:20.871 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:03:20.871 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:03:21.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:03:21.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:03:21.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:03:21.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-30T11:03:21.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-30T11:03:21.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:03:21.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-30T11:03:21.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:03:21.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:03:21.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:21.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-30T11:03:22.063 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-30T11:03:22.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:22.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-30T11:03:22.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:22.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:03:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:03:22.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:03:02.128907+0000 '>' 2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:22.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:03:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:03:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:03:23.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:03:23.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:03:23.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:03:23.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:03:23.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:03:23.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:03:02.128907+0000 '>' 2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:23.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:03:24.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:03:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:03:02.128907+0000 '>' 2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:03:25.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:03:25.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:03:25.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:03:25.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:03:25.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:03:25.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:03:25.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:03:26.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:03:02.128907+0000 '>' 2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:26.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:03:27.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:03:27.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:03:27.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:03:27.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:03:27.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:03:27.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:03:27.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:03:22.408376+0000 '>' 2024-03-30T11:03:02.128907+0000 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:03:27.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:03:27.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:03:27.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:03:27.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:03:27.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:03:27.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:03:27.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:03:27.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:03:27.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:03:27.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:03:28.990 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:03:28.991 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:03:28.991 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:03:29.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:03:29.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:03:29.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:03:29.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:03:29.264 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:03:29.265 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:03:29.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:03:29.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:03:29.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:03:29.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:03:29.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:03:29.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:03:29.293 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:29.295+0000 7fa3d3213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:29.295 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:29.298+0000 7fa3d3213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:29.296 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:29.299+0000 7fa3d3213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:29.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:03:29.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:03:29.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:03:29.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:03:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:03:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:03:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:03:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:03:29.812 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:29.815+0000 7fa3d3213640 -1 Falling back to public interface 2024-03-30T11:03:29.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:03:30.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:03:30.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:30.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:03:30.844 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:03:30.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:30.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:03:31.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:03:32.053 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:32.056+0000 7fa3d3213640 -1 osd.1 28 log_to_monitors true 2024-03-30T11:03:32.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:03:32.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:03:32.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:03:32.137 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:03:32.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:03:32.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:03:32.501 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 32 up_thru 27 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2837306035,v1:127.0.0.1:6811/2837306035] [v2:127.0.0.1:6812/2837306035,v1:127.0.0.1:6813/2837306035] exists,up 5f44bd1f-61b4-48c7-812e-08da329809b1 2024-03-30T11:03:32.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:03:32.502 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:03:32.503 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:03:32.503 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:03:32.503 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:03:32.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:03:32.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:03:32.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:03:32.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:03:32.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:03:32.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:03:32.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:03:32.888 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:03:32.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:03:32.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:03:32.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:03:33.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-30T11:03:33.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-30T11:03:33.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-03-30T11:03:33.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:03:33.096 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:03:33.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-30T11:03:33.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-30T11:03:33.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-03-30T11:03:33.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:03:33.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-03-30T11:03:33.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:03:33.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:03:33.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-03-30T11:03:33.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:03:33.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-30T11:03:33.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-03-30T11:03:33.258 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 111669149702 2024-03-30T11:03:33.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:33.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-03-30T11:03:33.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:03:34.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:03:34.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:34.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-03-30T11:03:34.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:03:35.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:03:35.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:03:36.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-30T11:03:36.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:03:36.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-30T11:03:36.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:03:36.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:03:36.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-30T11:03:36.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:03:36.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-30T11:03:36.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-30T11:03:36.219 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 137438953474 2024-03-30T11:03:36.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:03:36.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-30T11:03:36.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:03:36.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:03:36.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:03:36.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:03:36.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:03:36.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:03:36.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:03:36.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:03:36.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:03:36.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:03:36.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:03:37.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:03:37.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:03:37.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:03:37.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:03:37.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:03:37.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:03:37.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:03:37.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:03:37.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:03:37.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:03:37.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:03:37.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:03:37.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:03:37.628 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:03:37.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:03:37.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:03:37.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:03:37.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:03:37.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:03:37.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:03:37.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:03:37.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:03:37.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:03:37.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:03:37.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:03:37.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:03:37.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:03:37.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:03:37.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:03:37.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:03:37.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:03:37.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:03:37.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:03:37.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:03:37.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:03:37.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:03:37.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:03:37.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:03:37.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:03:37.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:03:37.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:03:37.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:03:37.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:03:37.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:03:37.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:03:37.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:03:37.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:03:37.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:03:37.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:03:37.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:03:37.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:03:37.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:03:37.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:03:37.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:03:37.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:03:37.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:03:37.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:03:37.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:03:37.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-30T11:03:37.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-30T11:03:37.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-30T11:03:39.239 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 0c806bbd-fe08-4cc0-9052-450daae2da18 2024-03-30T11:03:39.592 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:03:39.635 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.638+0000 7fe475fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:39.643 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.645+0000 7fe475fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:39.643 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.646+0000 7fe475fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:39.644 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.647+0000 7fe475fac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:03:39.644 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.647+0000 7fe475fac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:03:39.644 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.647+0000 7fe475fac640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:03:39.726 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:39.729+0000 7fe475fac640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:03:45.203 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:03:45.573 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:03:45.604 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:45.607+0000 7fad35a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:45.608 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:45.611+0000 7fad35a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:45.609 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:45.612+0000 7fad35a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:03:45.859 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:03:46.123 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:46.126+0000 7fad35a0d640 -1 Falling back to public interface 2024-03-30T11:03:47.141 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:03:48.432 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:03:48.451 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:03:48.454+0000 7fad35a0d640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:03:49.884 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:03:51.176 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:03:52.468 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:03:53.763 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:03:55.166 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:03:56.460 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:03:57.752 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:03:59.051 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:04:00.739 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:04:02.396 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:04:03.856 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:04:05.614 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:04:07.204 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:04:08.940 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:04:10.369 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:04:10.664 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/39857,v1:127.0.0.1:6803/39857] [v2:127.0.0.1:6804/39857,v1:127.0.0.1:6805/39857] exists,up 0c806bbd-fe08-4cc0-9052-450daae2da18 2024-03-30T11:04:10.667 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 e3c367fd-b275-4e91-ac94-8860b30c23de 2024-03-30T11:04:11.035 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:04:11.080 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.083+0000 7f577c381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.098+0000 7f577c381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.099+0000 7f577c381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.099+0000 7f577c381640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:04:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.099+0000 7f577c381640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:04:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.099+0000 7f577c381640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:04:11.162 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:11.165+0000 7f577c381640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:04:16.751 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:04:17.115 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:04:17.146 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:17.148+0000 7fdc4c4a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:17.148 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:17.150+0000 7fdc4c4a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:17.149 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:17.151+0000 7fdc4c4a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:17.396 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:04:17.671 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:17.672+0000 7fdc4c4a6640 -1 Falling back to public interface 2024-03-30T11:04:18.677 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:04:19.926 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:19.928+0000 7fdc4c4a6640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:04:19.969 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:04:21.411 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:04:22.835 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:04:24.128 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:04:25.505 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:04:26.869 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:04:28.160 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:04:29.514 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:04:30.820 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:04:32.114 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:04:33.407 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:04:34.702 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:04:36.313 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:04:37.949 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:04:39.660 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:04:41.346 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:04:41.707 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/435630588,v1:127.0.0.1:6811/435630588] [v2:127.0.0.1:6812/435630588,v1:127.0.0.1:6813/435630588] exists,up e3c367fd-b275-4e91-ac94-8860b30c23de 2024-03-30T11:04:41.710 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 55230bec-ab60-4110-9264-2db31f9f6840 2024-03-30T11:04:42.090 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:04:42.135 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.138+0000 7f356561e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:42.149 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.152+0000 7f356561e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.153+0000 7f356561e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.153+0000 7f356561e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:04:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.153+0000 7f356561e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:04:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.153+0000 7f356561e640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:04:42.241 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:42.244+0000 7f356561e640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:04:47.989 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:04:48.401 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:04:48.432 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:48.435+0000 7f8d44a22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:48.434 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:48.437+0000 7f8d44a22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:48.435 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:48.438+0000 7f8d44a22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:04:48.686 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:04:48.963 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:48.966+0000 7f8d44a22640 -1 Falling back to public interface 2024-03-30T11:04:49.965 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:04:51.257 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:04:51.290 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:04:51.293+0000 7f8d44a22640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:04:52.623 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:04:53.916 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:04:55.285 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:04:56.577 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:04:57.939 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:04:59.281 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:05:00.572 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:05:01.938 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:05:03.281 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:05:04.846 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:05:06.375 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:05:07.884 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:05:09.564 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:05:11.119 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:05:12.583 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:05:12.952 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2535170375,v1:127.0.0.1:6819/2535170375] [v2:127.0.0.1:6820/2535170375,v1:127.0.0.1:6821/2535170375] exists,up 55230bec-ab60-4110-9264-2db31f9f6840 2024-03-30T11:05:13.385 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T11:05:15.249 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836497 2024-03-30T11:05:16.824 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672969 2024-03-30T11:05:17.107 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509447 2024-03-30T11:05:19.700 INFO:tasks.workunit.client.0.smithi003.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-03-30T11:05:19.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-03-30T11:05:19.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-03-30T11:05:19.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-03-30T11:05:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 2 2024-03-30T11:05:20.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-30T11:05:20.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:05:20.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:05:20.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:05:20.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:05:20.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:05:20.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:05:20.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:05:20.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:05:20.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:05:20.626 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:05:20.626 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:05:20.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:05:20.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:20.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:05:20.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-30T11:05:20.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-30T11:05:20.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-30T11:05:20.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:20.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:05:20.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-30T11:05:20.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-30T11:05:20.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-03-30T11:05:20.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:20.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:05:20.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-30T11:05:20.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-30T11:05:20.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971 2-64424509449' 2024-03-30T11:05:20.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:20.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-30T11:05:20.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:20.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:05:20.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-30T11:05:20.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:20.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-30T11:05:20.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-30T11:05:20.952 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836499 2024-03-30T11:05:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:21.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836499 2024-03-30T11:05:21.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:05:22.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:05:22.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:22.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:22.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-30T11:05:22.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-30T11:05:22.536 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672971 2024-03-30T11:05:22.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:05:22.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672971 2024-03-30T11:05:22.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:22.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-30T11:05:22.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:22.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:05:22.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-30T11:05:22.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:22.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-30T11:05:22.823 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509449 2024-03-30T11:05:22.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-30T11:05:22.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:05:23.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-30T11:05:23.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:05:23.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:23.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:23.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:05:23.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:23.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:23.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:23.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:23.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:23.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:23.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:23.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:05:23.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:23.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:23.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:05:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:05:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:05:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-03-30T11:05:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-03-30T11:05:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:24.205 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:05:24.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:05:24.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:24.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:05:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:05:24.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:05:25.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:05:25.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-30T11:05:26.370 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-30T11:05:26.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:05:26.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:05:26.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:05:26.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:05:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:05:26.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:05:26.890 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:05:26.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:05:26.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:05:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:05:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:05:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:05:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:05:26.917 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:26.919+0000 7f1e5e80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:05:26.919 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:26.921+0000 7f1e5e80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:05:26.920 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:26.923+0000 7f1e5e80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:05:27.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:27.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:27.434 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:27.437+0000 7f1e5e80d640 -1 Falling back to public interface 2024-03-30T11:05:27.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:05:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:05:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:05:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:05:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:28.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:28.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:05:29.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:05:29.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:29.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:05:29.764 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:05:29.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:29.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:29.785 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:29.787+0000 7f1e5e80d640 -1 osd.0 21 log_to_monitors true 2024-03-30T11:05:30.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:05:31.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:05:31.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:31.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:05:31.069 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:05:31.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:31.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4132597268,v1:127.0.0.1:6803/4132597268] [v2:127.0.0.1:6804/4132597268,v1:127.0.0.1:6805/4132597268] exists,up 0c806bbd-fe08-4cc0-9052-450daae2da18 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:05:31.360 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:05:31.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:05:31.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:05:31.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:05:31.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:05:31.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:05:31.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:05:31.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:05:31.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:05:31.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:05:31.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:05:31.752 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:05:31.752 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:05:31.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:05:31.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:31.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:05:31.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-30T11:05:31.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-30T11:05:31.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-30T11:05:31.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:31.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:05:32.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-30T11:05:32.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-30T11:05:32.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-03-30T11:05:32.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:32.001 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:05:32.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-30T11:05:32.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-30T11:05:32.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975 2-64424509452' 2024-03-30T11:05:32.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:32.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-30T11:05:32.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:32.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:05:32.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-30T11:05:32.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:32.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-30T11:05:32.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-30T11:05:32.109 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 107374182402 2024-03-30T11:05:32.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:32.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-30T11:05:32.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:05:33.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:05:33.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:33.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-30T11:05:33.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:33.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-30T11:05:33.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:33.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:05:33.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-30T11:05:33.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:33.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-30T11:05:33.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-30T11:05:33.686 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672975 2024-03-30T11:05:33.686 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:05:33.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-30T11:05:33.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:33.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-30T11:05:33.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:33.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:05:33.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-30T11:05:33.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:33.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-30T11:05:33.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-30T11:05:33.970 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509452 2024-03-30T11:05:33.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:05:34.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-30T11:05:34.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:05:34.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:34.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:34.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:34.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:05:34.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:34.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:34.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:35.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:05:35.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:05:35.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:05:35.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-30T11:05:35.333 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0: { 2024-03-30T11:05:35.333 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-30T11:05:35.333 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:05:35.347 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1: { 2024-03-30T11:05:35.347 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-30T11:05:35.347 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:05:35.361 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2: { 2024-03-30T11:05:35.361 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-30T11:05:35.361 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:05:35.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-03-30T11:05:35.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T11:05:35.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:05:35.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:35.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:05:35.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-03-30T11:05:35.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-03-30T11:05:35.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:05:35.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:05:35.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:35.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:05:35.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-03-30T11:05:35.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-03-30T11:05:35.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:05:35.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:05:35.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:05:35.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:05:36.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:36.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-03-30T11:05:36.319 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:05:36.319 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": true, 2024-03-30T11:05:36.319 INFO:tasks.workunit.client.0.smithi003.stdout: "must": false, 2024-03-30T11:05:36.319 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "2024-03-16T11:03:56.323068+0000" 2024-03-30T11:05:36.319 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:05:36.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:36.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T11:05:36.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:36.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:05:36.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:05:36.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:05:36.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:05:36.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:05:36.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:05:36.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:05:36.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:05:36.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:05:13.271951+0000 '>' 2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:36.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:05:37.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:05:37.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:05:37.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:05:37.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:05:37.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:05:37.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:05:37.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:05:37.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-16T11:03:56.323068+0000 '>' 2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:37.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:05:38.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:05:38.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:05:38.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:05:38.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:05:38.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:05:38.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:05:38.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:05:39.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-16T11:03:56.323068+0000 '>' 2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:39.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:05:40.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:05:40.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:05:40.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:05:40.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:05:40.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:05:40.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:05:40.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:05:40.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-16T11:03:56.323068+0000 '>' 2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:40.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:05:41.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:05:41.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:05:41.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:05:41.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:05:41.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:05:41.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:05:41.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:05:36.491807+0000 '>' 2024-03-30T11:05:13.271951+0000 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:05:41.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:05:41.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:05:41.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:05:41.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:05:41.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:05:41.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:05:41.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:05:41.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:05:41.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:05:42.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:05:42.189 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:05:42.189 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:05:42.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:05:42.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:42.189 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:05:42.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-30T11:05:42.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-30T11:05:42.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-30T11:05:42.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:42.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:05:42.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-30T11:05:42.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-30T11:05:42.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672978' 2024-03-30T11:05:42.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:42.399 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:05:42.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-30T11:05:42.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-30T11:05:42.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672978 2-64424509455' 2024-03-30T11:05:42.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:42.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-30T11:05:42.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:42.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:05:42.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-30T11:05:42.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:42.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-30T11:05:42.511 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 107374182405 2024-03-30T11:05:42.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-30T11:05:42.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:42.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-03-30T11:05:42.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:05:43.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:05:43.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:44.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-30T11:05:44.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:44.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-30T11:05:44.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:44.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:05:44.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-30T11:05:44.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:44.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-30T11:05:44.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-30T11:05:44.087 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672978 2024-03-30T11:05:44.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:05:44.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-30T11:05:44.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:44.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-30T11:05:44.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:44.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:05:44.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-30T11:05:44.380 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:44.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-30T11:05:44.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-30T11:05:44.381 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509455 2024-03-30T11:05:44.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:05:44.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-30T11:05:44.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:05:44.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:44.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:05:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:45.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:45.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:45.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:45.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:45.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:45.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:45.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:05:45.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:45.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:45.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:45.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:05:45.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:05:45.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:05:45.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-03-30T11:05:45.948 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:05:45.948 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-30T11:05:36.637003+0000 21'1 26:58 [1,0] 1 [1,0] 1 21'1 2024-03-30T11:05:36.491807+0000 21'1 2024-03-30T11:05:36.491807+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:05:36.491807+0000 1 0 2024-03-30T11:05:45.948 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:05:45.948 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:05:45.949 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:05:45.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:45.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:05:46.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:05:46.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:46.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:05:46.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:05:46.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:05:46.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:05:46.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-30T11:05:48.007 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:05:48.007 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:05:48.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:05:48.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:05:48.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:05:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:05:48.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:05:48.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:05:48.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:05:48.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:05:48.277 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:05:48.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:05:48.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:05:48.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:05:48.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:05:48.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:05:48.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:05:48.304 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:48.307+0000 7f6da8eb3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:05:48.326 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:48.330+0000 7f6da8eb3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:05:48.327 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:48.331+0000 7f6da8eb3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:48.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:48.840 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:48.844+0000 7f6da8eb3640 -1 Falling back to public interface 2024-03-30T11:05:48.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:05:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:05:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:05:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:05:49.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:49.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:50.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:05:51.105 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:05:51.109+0000 7f6da8eb3640 -1 osd.0 26 log_to_monitors true 2024-03-30T11:05:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:05:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:05:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:05:51.180 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:05:51.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:05:51.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3343455811,v1:127.0.0.1:6803/3343455811] [v2:127.0.0.1:6804/3343455811,v1:127.0.0.1:6805/3343455811] exists,up 0c806bbd-fe08-4cc0-9052-450daae2da18 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:05:51.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:05:51.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:05:51.537 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:05:51.537 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:05:51.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:05:51.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:05:51.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:05:51.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:05:51.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:05:51.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:05:51.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:05:51.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:05:51.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:05:51.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:05:51.903 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:05:51.903 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:05:51.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:05:51.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:51.904 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:05:52.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-30T11:05:52.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-30T11:05:52.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-30T11:05:52.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:52.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:05:52.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-30T11:05:52.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-30T11:05:52.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981' 2024-03-30T11:05:52.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:05:52.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:05:52.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-30T11:05:52.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-30T11:05:52.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672981 2-64424509458' 2024-03-30T11:05:52.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:52.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-30T11:05:52.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:52.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:05:52.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-30T11:05:52.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:52.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-30T11:05:52.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-30T11:05:52.220 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 128849018882 2024-03-30T11:05:52.220 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:52.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-30T11:05:52.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:05:53.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:05:53.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:05:53.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-30T11:05:53.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:53.816 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-30T11:05:53.816 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:53.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:05:53.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-30T11:05:53.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:53.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-30T11:05:53.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-30T11:05:53.818 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672981 2024-03-30T11:05:53.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:05:54.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-30T11:05:54.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:05:54.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-30T11:05:54.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:05:54.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:05:54.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-30T11:05:54.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:05:54.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-30T11:05:54.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-30T11:05:54.106 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509458 2024-03-30T11:05:54.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:05:54.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-30T11:05:54.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:05:54.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:54.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:54.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:05:54.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:54.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:54.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:54.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:54.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:54.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:54.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:55.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:05:55.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:55.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:55.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:55.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:05:55.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-30T11:05:55.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:05:55.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-30T11:05:55.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:05:55.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:05:55.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:05:55.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:55.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:55.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:55.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:55.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:55.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:55.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:55.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:05:55.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:55.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:55.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:56.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:05:56.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:05:56.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:05:56.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:05:56.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:05:56.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:05:56.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:05:56.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:05:56.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:05:56.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:05:56.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:05:56.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:05:56.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:05:56.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:05:56.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:05:56.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:56.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:56.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:56.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:56.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:56.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:56.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:57.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:05:57.002 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:57.002 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:57.002 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:57.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:05:57.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:05:57.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:05:57.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:05:57.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:05:57.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T11:05:57.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:05:57.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T11:05:58.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:05:58.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:05:58.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:05:58.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:05:58.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:05:58.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:05:58.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:05:58.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:05:58.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:05:58.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:05:58.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:05:58.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:58.741 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:05:59.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:05:59.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:05:59.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:05:59.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:05:59.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:05:59.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:05:59.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:05:59.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:05:59.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:06:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:06:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:06:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:06:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:06:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:06:01.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:06:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:06:01.253 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:06:01.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:06:01.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:06:01.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:06:01.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:06:01.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:06:01.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:06:01.279 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:01.282+0000 7ff734f8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:01.281 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:01.284+0000 7ff734f8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:01.282 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:01.286+0000 7ff734f8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:01.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:01.796 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:01.800+0000 7ff734f8c640 -1 Falling back to public interface 2024-03-30T11:06:01.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:02.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:02.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:02.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:06:02.826 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:06:02.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:02.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:03.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:04.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:04.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:04.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:06:04.120 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:06:04.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:04.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:04.202 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:04.205+0000 7ff734f8c640 -1 osd.0 31 log_to_monitors true 2024-03-30T11:06:04.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:05.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:05.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:05.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:06:05.491 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:06:05.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:05.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/1712616952,v1:127.0.0.1:6803/1712616952] [v2:127.0.0.1:6804/1712616952,v1:127.0.0.1:6805/1712616952] exists,up 0c806bbd-fe08-4cc0-9052-450daae2da18 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:06:05.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:06:05.840 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:06:05.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:06:05.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:06:05.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:06:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:06:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:06:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:06:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:06:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:06:05.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:06:06.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:06:06.201 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:06:06.201 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:06:06.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:06:06.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:06:06.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:06:06.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-30T11:06:06.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-30T11:06:06.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-30T11:06:06.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:06:06.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:06:06.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-30T11:06:06.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-30T11:06:06.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672984' 2024-03-30T11:06:06.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:06:06.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:06:06.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-30T11:06:06.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-30T11:06:06.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672984 2-64424509462' 2024-03-30T11:06:06.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:06:06.573 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-30T11:06:06.573 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:06:06.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:06:06.574 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-30T11:06:06.574 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:06:06.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-30T11:06:06.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-30T11:06:06.575 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 150323855362 2024-03-30T11:06:06.575 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:06:06.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-30T11:06:06.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:06:07.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:06:07.862 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:06:08.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-30T11:06:08.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:06:08.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-30T11:06:08.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:06:08.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:06:08.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-30T11:06:08.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:06:08.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-30T11:06:08.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-30T11:06:08.157 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672984 2024-03-30T11:06:08.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:06:08.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672984 2024-03-30T11:06:08.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:06:08.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-30T11:06:08.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:06:08.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:06:08.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-30T11:06:08.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:06:08.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-30T11:06:08.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-30T11:06:08.447 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509462 2024-03-30T11:06:08.448 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:06:08.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-30T11:06:08.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:06:08.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:06:08.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:06:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:06:09.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:06:09.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:06:09.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:06:09.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:06:09.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:06:09.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:06:09.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:06:09.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:06:09.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-30T11:06:09.714 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:05:36.490+0000 7fdc3041f640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:06:09.714 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:05:36.490+0000 7fdc3041f640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:06:09.714 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:05:36.490+0000 7fdc3041f640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-30T11:06:09.714 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:05:36.490+0000 7fdc3041f640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-30T11:06:09.714 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:05:36.490+0000 7fdc3041f640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:06:09.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:06:09.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:06:09.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:06:09.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:06:09.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:06:09.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:06:09.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:06:09.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:06:09.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:06:09.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:06:09.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:06:09.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:06:09.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:06:09.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:06:09.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:06:09.825 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:06:09.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:06:09.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:06:09.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:06:09.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:06:09.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:06:09.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:06:09.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:06:09.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:06:09.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:09.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:09.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:06:09.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:06:09.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:06:09.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:06:09.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:06:09.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:06:09.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:06:09.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:06:09.847 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:06:09.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:06:09.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:06:09.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:06:09.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:06:09.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:06:09.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:06:09.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:06:09.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:06:09.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:06:09.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:06:09.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:06:09.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:09.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:09.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:06:09.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:06:09.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:06:09.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:06:09.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:06:09.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:09.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:09.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:06:09.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:06:09.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:06:09.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:06:09.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:06:09.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-03-30T11:06:09.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:06:09.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:06:10.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:06:10.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:06:10.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:06:10.233 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:06:10.233 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:10.233 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:10.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:10.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:06:10.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:06:10.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:06:10.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:06:10.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:06:10.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:06:10.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:06:10.427 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:06:10.427 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:06:10.427 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:06:10.428 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:06:10.428 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:10.428 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:10.428 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:06:10.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:06:10.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:06:10.512 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:06:10.513 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:10.513 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:10.513 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:06:10.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:06:10.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:06:10.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-03-30T11:06:10.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:06:10.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:06:10.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:06:10.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:06:10.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:06:10.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:06:11.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:11.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:06:11.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:06:11.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:06:11.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:06:11.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:11.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:06:11.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:06:11.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:06:11.446 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:06:11.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9f2deed3-8219-4be2-9058-f2a179463e93' 2024-03-30T11:06:11.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:06:11.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAj8gdml7nAGxAAbRUP7C46TajDpgJFj1jOYg== 2024-03-30T11:06:11.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAj8gdml7nAGxAAbRUP7C46TajDpgJFj1jOYg=="}' 2024-03-30T11:06:11.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f2deed3-8219-4be2-9058-f2a179463e93 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:06:11.682 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:06:11.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:06:11.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAj8gdml7nAGxAAbRUP7C46TajDpgJFj1jOYg== --osd-uuid 9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:06:11.725 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.726+0000 7f6323bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:11.733 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.737+0000 7f6323bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:11.734 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.738+0000 7f6323bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:11.734 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.738+0000 7f6323bff640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:06:11.735 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.738+0000 7f6323bff640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:06:11.736 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.738+0000 7f6323bff640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:06:11.825 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:11.828+0000 7f6323bff640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:06:17.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:06:17.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:06:17.188 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:06:17.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:06:17.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:06:17.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:06:17.582 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:06:17.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:06:17.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:06:17.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:06:17.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:06:17.611 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:17.615+0000 7fe08b8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:17.613 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:17.617+0000 7fe08b8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:17.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:17.618+0000 7fe08b8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:17.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:06:17.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:06:17.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:06:17.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:06:17.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:06:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:06:17.867 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:06:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:18.129 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:18.133+0000 7fe08b8d9640 -1 Falling back to public interface 2024-03-30T11:06:18.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:19.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:19.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:19.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:06:19.144 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:06:19.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:19.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:19.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:20.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:20.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:20.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:06:20.437 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:06:20.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:20.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:20.480 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:20.483+0000 7fe08b8d9640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:06:20.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:21.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:21.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:21.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:06:21.746 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:06:21.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:21.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:23.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:23.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:23.044 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:06:23.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:06:23.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:23.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:23.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:24.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:24.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:24.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:06:24.398 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:06:24.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:24.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:24.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:25.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:25.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:25.690 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:06:25.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:06:25.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:25.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:25.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:26.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:26.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:27.000 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:06:27.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:06:27.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:27.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:27.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:28.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:28.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:28.292 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:06:28.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:06:28.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:28.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:28.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:29.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:29.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:29.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:06:29.584 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:06:29.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:29.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:29.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:30.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:30.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:30.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:06:30.879 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:06:30.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:30.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:31.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:32.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:32.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:32.759 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:06:32.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:06:32.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:32.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:33.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:34.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:34.396 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:06:34.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:34.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:06:34.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:34.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:34.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:35.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:35.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:35.848 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:06:35.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:06:35.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:35.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:36.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:37.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:37.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:37.184 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:06:37.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:06:37.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:37.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:37.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:38.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:38.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:38.618 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:06:38.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:06:38.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:38.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:39.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:40.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:40.138 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:06:40.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:40.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:06:40.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:40.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:40.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:41.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:41.753 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:06:41.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:41.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:06:41.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:41.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:06:42.128 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2463385288,v1:127.0.0.1:6803/2463385288] [v2:127.0.0.1:6804/2463385288,v1:127.0.0.1:6805/2463385288] exists,up 9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:06:42.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:06:42.133 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:06:42.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:06:42.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:06:42.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:06:42.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f6819c6-b233-4910-abda-a0a5ff413b05 2024-03-30T11:06:42.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7f6819c6-b233-4910-abda-a0a5ff413b05' 2024-03-30T11:06:42.135 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 7f6819c6-b233-4910-abda-a0a5ff413b05 2024-03-30T11:06:42.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:06:42.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBC8gdmUXchCRAARMdUJLqW8lke8d1TqclCbg== 2024-03-30T11:06:42.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBC8gdmUXchCRAARMdUJLqW8lke8d1TqclCbg=="}' 2024-03-30T11:06:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f6819c6-b233-4910-abda-a0a5ff413b05 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:06:42.509 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:06:42.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:06:42.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBC8gdmUXchCRAARMdUJLqW8lke8d1TqclCbg== --osd-uuid 7f6819c6-b233-4910-abda-a0a5ff413b05 2024-03-30T11:06:42.551 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.553+0000 7fe8a4ef9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:42.593 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.596+0000 7fe8a4ef9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:42.594 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.597+0000 7fe8a4ef9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:42.594 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.597+0000 7fe8a4ef9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:06:42.595 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.597+0000 7fe8a4ef9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:06:42.595 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.597+0000 7fe8a4ef9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:06:42.668 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:42.671+0000 7fe8a4ef9640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:06:48.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:06:48.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:06:48.570 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:06:48.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:06:48.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:06:48.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:06:48.942 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:06:48.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:06:48.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:06:48.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:06:48.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:06:48.970 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:48.974+0000 7f873e068640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:48.972 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:48.976+0000 7f873e068640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:48.973 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:48.977+0000 7f873e068640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:06:49.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:06:49.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:06:49.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:06:49.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:06:49.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:06:49.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:49.227 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:06:49.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:06:49.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:49.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:49.491 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:49.495+0000 7f873e068640 -1 Falling back to public interface 2024-03-30T11:06:49.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:50.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:50.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:50.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:06:50.511 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:06:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:50.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:51.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:51.801 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:06:51.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:51.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:06:51.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:51.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:51.874 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:06:51.878+0000 7f873e068640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:06:52.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:53.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:53.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:53.095 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:06:53.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:06:53.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:53.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:53.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:54.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:54.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:54.387 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:06:54.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:06:54.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:54.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:55.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:55.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:55.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:06:55.703 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:06:55.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:55.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:56.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:57.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:57.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:57.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:06:57.000 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:06:57.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:57.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:57.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:58.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:58.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:58.294 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:06:58.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:06:58.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:58.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:06:59.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:06:59.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:06:59.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:06:59.590 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:06:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:06:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:06:59.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:00.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:00.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:00.887 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:07:00.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:07:00.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:00.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:01.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:02.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:02.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:02.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:07:02.236 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:07:02.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:02.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:02.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:03.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:03.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:03.530 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:07:03.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:07:03.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:03.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:04.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:05.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:05.405 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:07:05.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:05.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:07:05.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:05.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:06.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:07.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:07.082 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:07:07.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:07.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:07:07.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:07.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:07.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:08.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:08.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:08.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:07:08.704 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:07:08.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:08.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:09.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:10.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:10.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:10.451 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:07:10.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:07:10.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:10.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:11.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:12.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:12.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:12.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:07:12.022 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:07:12.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:12.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:12.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:13.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:13.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:13.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:07:13.840 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:07:13.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:13.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:14.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:15.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:15.174 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:07:15.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:15.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:07:15.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:15.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:07:15.465 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1356417049,v1:127.0.0.1:6811/1356417049] [v2:127.0.0.1:6812/1356417049,v1:127.0.0.1:6813/1356417049] exists,up 7f6819c6-b233-4910-abda-a0a5ff413b05 2024-03-30T11:07:15.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:07:15.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:07:15.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:07:15.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:07:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=218da1c9-8981-41ca-b680-698c93b583b9 2024-03-30T11:07:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 218da1c9-8981-41ca-b680-698c93b583b9' 2024-03-30T11:07:15.473 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 218da1c9-8981-41ca-b680-698c93b583b9 2024-03-30T11:07:15.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:07:15.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBj8gdmwUc9HRAAUBBPCpGkoV1UfHgUUfJXIA== 2024-03-30T11:07:15.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBj8gdmwUc9HRAAUBBPCpGkoV1UfHgUUfJXIA=="}' 2024-03-30T11:07:15.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 218da1c9-8981-41ca-b680-698c93b583b9 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:07:15.885 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:07:15.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:07:15.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBj8gdmwUc9HRAAUBBPCpGkoV1UfHgUUfJXIA== --osd-uuid 218da1c9-8981-41ca-b680-698c93b583b9 2024-03-30T11:07:15.927 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:15.931+0000 7f6803e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:07:15.970 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:15.974+0000 7f6803e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:07:15.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:15.975+0000 7f6803e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:07:15.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:15.975+0000 7f6803e0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:07:15.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:15.975+0000 7f6803e0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:07:15.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:15.975+0000 7f6803e0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:07:16.036 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:16.040+0000 7f6803e0f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:07:21.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:07:21.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:07:21.968 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:07:21.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:07:21.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:07:22.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:07:22.332 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:07:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:07:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:07:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:07:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:07:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:22.364+0000 7f4b36415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:07:22.362 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:22.366+0000 7f4b36415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:07:22.363 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:22.367+0000 7f4b36415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:22.622 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:07:22.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:07:22.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:22.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:22.878 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:22.882+0000 7f4b36415640 -1 Falling back to public interface 2024-03-30T11:07:22.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:23.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:23.904 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:07:23.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:23.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:07:23.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:23.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:24.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:25.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:25.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:25.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:07:25.198 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:07:25.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:25.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:25.210 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:07:25.214+0000 7f4b36415640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:07:25.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:26.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:26.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:26.629 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:07:26.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:07:26.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:26.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:26.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:27.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:27.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:27.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:07:27.956 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:07:27.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:27.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:28.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:29.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:29.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:29.248 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:07:29.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:07:29.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:29.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:29.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:30.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:30.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:30.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:07:30.546 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:07:30.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:30.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:30.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:31.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:31.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:31.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:07:31.910 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:07:31.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:31.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:32.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:33.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:33.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:33.203 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:07:33.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:07:33.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:33.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:33.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:34.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:34.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:34.498 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:07:34.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:07:34.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:34.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:34.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:35.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:35.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:35.882 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:07:35.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:07:35.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:35.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:36.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:37.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:37.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:37.172 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:07:37.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:07:37.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:37.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:37.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:38.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:38.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:38.727 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:07:38.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:07:38.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:38.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:39.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:40.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:40.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:40.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:07:40.184 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:07:40.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:40.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:40.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:41.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:41.741 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:07:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:07:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:41.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:42.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:43.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:43.037 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:07:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:07:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:43.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:43.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:44.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:44.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:44.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:07:44.856 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:07:44.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:44.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:45.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:46.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:46.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:46.159 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:07:46.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:07:46.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:46.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:46.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:07:47.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:07:47.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:07:47.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:07:47.553 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:07:47.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:07:47.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:07:47.850 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3863233013,v1:127.0.0.1:6819/3863233013] [v2:127.0.0.1:6820/3863233013,v1:127.0.0.1:6821/3863233013] exists,up 218da1c9-8981-41ca-b680-698c93b583b9 2024-03-30T11:07:47.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:07:47.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:07:47.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:07:47.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-03-30T11:07:47.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-30T11:07:48.222 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T11:07:48.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:07:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-03-30T11:07:49.887 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 2 2024-03-30T11:07:49.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-30T11:07:49.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:07:49.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:07:49.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:07:49.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:07:49.901 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:07:49.901 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:07:49.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:07:49.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:07:49.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:07:49.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:07:49.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:07:49.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:07:49.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:07:49.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:07:49.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:07:50.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:07:50.421 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:07:50.421 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:07:50.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:07:50.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:07:50.422 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:07:50.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-30T11:07:50.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-30T11:07:50.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-30T11:07:50.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:07:50.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:07:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-30T11:07:50.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-30T11:07:50.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-03-30T11:07:50.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:07:50.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:07:50.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-30T11:07:50.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-30T11:07:50.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509445' 2024-03-30T11:07:50.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:07:50.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-30T11:07:50.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:07:50.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:07:50.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-30T11:07:50.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:07:50.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-30T11:07:50.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-30T11:07:50.892 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836498 2024-03-30T11:07:50.892 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:07:51.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-30T11:07:51.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:07:52.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:07:52.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:07:52.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-30T11:07:52.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:07:52.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-30T11:07:52.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:07:52.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:07:52.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-30T11:07:52.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:07:52.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-30T11:07:52.566 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672976 2024-03-30T11:07:52.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-30T11:07:52.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:07:52.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-30T11:07:52.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:07:52.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-30T11:07:52.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:07:52.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:07:52.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-30T11:07:52.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:07:52.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-30T11:07:52.852 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509445 2024-03-30T11:07:52.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-30T11:07:52.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:07:53.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-30T11:07:53.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:07:53.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:07:53.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:07:53.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:07:53.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:07:53.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:07:53.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:07:53.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:07:53.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:07:53.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:07:53.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:07:53.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:07:53.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:07:53.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:07:53.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:07:54.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:07:54.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-30T11:07:54.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:07:54.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-30T11:07:54.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:07:54.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:07:54.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:07:54.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:07:54.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:07:54.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:07:54.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:07:54.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:07:55.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:07:55.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:07:55.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:07:55.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:07:55.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:07:55.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:07:55.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:07:56.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:07:56.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:07:56.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:07:56.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-03-30T11:07:56.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-03-30T11:07:56.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-03-30T11:07:56.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:56.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:56.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:57.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:57.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:57.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-30T11:07:57.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:07:57.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:07:57.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:07:57.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:07:57.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:07:57.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:07:57.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:07:58.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-30T11:07:59.544 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-30T11:08:00.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:00.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:08:00.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:08:00.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:08:00.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:08:00.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:08:00.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:08:00.068 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:08:00.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:08:00.070 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:08:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:00.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:08:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:08:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:08:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:08:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:08:00.098 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:00.101+0000 7f28a3268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:00.100 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:00.103+0000 7f28a3268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:00.101 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:00.104+0000 7f28a3268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:00.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:08:00.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:00.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:00.617 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:00.621+0000 7f28a3268640 -1 Falling back to public interface 2024-03-30T11:08:00.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:01.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:01.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:01.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:08:01.652 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:01.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:01.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:01.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:02.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:02.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:02.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:08:02.945 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:08:02.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:02.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:03.010 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:03.014+0000 7f28a3268640 -1 osd.0 21 log_to_monitors true 2024-03-30T11:08:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:04.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:04.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:04.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:08:04.290 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:08:04.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:04.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4098378531,v1:127.0.0.1:6803/4098378531] [v2:127.0.0.1:6804/4098378531,v1:127.0.0.1:6805/4098378531] exists,up 9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:08:04.622 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:08:04.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:08:04.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:08:04.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:08:04.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:08:04.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:08:04.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:08:04.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:08:04.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:08:04.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:08:04.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:08:04.985 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:04.985 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:08:04.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:08:04.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:04.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:08:05.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-30T11:08:05.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-30T11:08:05.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-30T11:08:05.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:05.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:08:05.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-30T11:08:05.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-30T11:08:05.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-03-30T11:08:05.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:05.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:08:05.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-30T11:08:05.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-30T11:08:05.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509449' 2024-03-30T11:08:05.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:05.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-30T11:08:05.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:05.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:08:05.302 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-30T11:08:05.302 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:05.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-30T11:08:05.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-30T11:08:05.303 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 107374182402 2024-03-30T11:08:05.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:05.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-30T11:08:05.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:06.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:08:06.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:06.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-30T11:08:06.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:06.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-30T11:08:06.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:06.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:08:06.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-30T11:08:06.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-30T11:08:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-30T11:08:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672980 2024-03-30T11:08:06.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-30T11:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-30T11:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:07.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:08:07.169 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-30T11:08:07.170 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:07.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-30T11:08:07.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-30T11:08:07.171 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509449 2024-03-30T11:08:07.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:08:07.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-30T11:08:07.451 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:08:07.451 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:07.451 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:07.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:08:07.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:08:07.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:08:07.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:08:07.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:08:07.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:08:07.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:08:07.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:08:08.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:08:08.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:08:08.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:08.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:08:08.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:08:08.434 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:08:08.434 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:08:08.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:08:08.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:08:08.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:08:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:08:09.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:08:09.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:08:09.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:08:09.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:08:09.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:08:10.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:08:10.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-30T11:08:11.298 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:104778fc:::obj2:head# 2024-03-30T11:08:11.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:08:11.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:08:11.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:08:11.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:08:11.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:08:11.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:08:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:11.851+0000 7faa84a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:11.850 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:11.854+0000 7faa84a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:11.852 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:11.856+0000 7faa84a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:12.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:08:12.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:08:12.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:08:12.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:08:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:08:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:08:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:08:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:12.373 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:12.378+0000 7faa84a13640 -1 Falling back to public interface 2024-03-30T11:08:12.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:13.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:13.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:13.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:08:13.398 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:13.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:13.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:13.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:14.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:14.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:14.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:08:14.691 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:08:14.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:14.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:14.792 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:14.796+0000 7faa84a13640 -1 osd.0 27 log_to_monitors true 2024-03-30T11:08:15.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:16.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:16.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:16.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:08:16.053 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:08:16.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:16.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,27) [v2:127.0.0.1:6802/484687290,v1:127.0.0.1:6803/484687290] [v2:127.0.0.1:6804/484687290,v1:127.0.0.1:6805/484687290] exists,up 9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:08:16.349 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:08:16.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:08:16.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:08:16.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:08:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:08:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:08:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:08:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:08:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:08:16.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:08:16.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:08:16.750 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:16.750 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:08:16.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:08:16.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:16.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:08:16.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-30T11:08:16.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-30T11:08:16.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-30T11:08:16.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:16.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:08:16.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-30T11:08:16.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-30T11:08:16.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983' 2024-03-30T11:08:16.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:16.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:08:17.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-30T11:08:17.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-30T11:08:17.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983 2-64424509452' 2024-03-30T11:08:17.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:17.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-30T11:08:17.066 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:17.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:08:17.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-30T11:08:17.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:17.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-30T11:08:17.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-30T11:08:17.068 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 128849018882 2024-03-30T11:08:17.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:17.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-30T11:08:17.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:18.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:08:18.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:18.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-30T11:08:18.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:18.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-30T11:08:18.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672983 2024-03-30T11:08:18.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:18.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-30T11:08:18.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:18.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-30T11:08:18.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:18.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:08:18.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-30T11:08:18.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:18.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-30T11:08:18.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-30T11:08:18.935 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509452 2024-03-30T11:08:18.935 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:08:19.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-30T11:08:19.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:08:19.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:19.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:08:19.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:08:19.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:08:19.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:08:19.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:19.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:20.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:08:20.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:08:20.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:08:20.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-30T11:08:20.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:08:20.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:08:20.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:08:20.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:08:20.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-30T11:08:20.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:08:20.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:08:20.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:08:20.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:08:20.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:08:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:08:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-30T11:08:22.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:08:22.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:08:22.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:08:22.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:22.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:08:22.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:08:22.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:08:22.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:08:22.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:08:22.609 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:22.613+0000 7ff52eed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:22.611 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:22.615+0000 7ff52eed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:22.612 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:22.617+0000 7ff52eed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:22.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:08:22.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:22.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:08:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:23.139+0000 7ff52eed3640 -1 Falling back to public interface 2024-03-30T11:08:23.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:24.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:24.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:24.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:08:24.153 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:24.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:24.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:08:24.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:25.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:25.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:25.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:08:25.446 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:08:25.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:25.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:08:25.499 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:25.503+0000 7ff52eed3640 -1 osd.1 31 log_to_monitors true 2024-03-30T11:08:25.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:26.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:26.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:26.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:08:26.755 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:08:26.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:26.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:08:27.045 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/3878793872,v1:127.0.0.1:6811/3878793872] [v2:127.0.0.1:6812/3878793872,v1:127.0.0.1:6813/3878793872] exists,up 7f6819c6-b233-4910-abda-a0a5ff413b05 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:08:27.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:08:27.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:08:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:08:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:08:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:08:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:08:27.129 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:08:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:08:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:08:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:08:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:27.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:08:27.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-30T11:08:27.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-30T11:08:27.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-03-30T11:08:27.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:27.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:08:27.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-30T11:08:27.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-30T11:08:27.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-03-30T11:08:27.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:27.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:08:27.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-30T11:08:27.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-30T11:08:27.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509456' 2024-03-30T11:08:27.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:27.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-03-30T11:08:27.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:27.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:08:27.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-03-30T11:08:27.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:27.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-30T11:08:27.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-03-30T11:08:27.727 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 128849018885 2024-03-30T11:08:27.727 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:28.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-30T11:08:28.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:28.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-03-30T11:08:28.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:28.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:08:28.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-03-30T11:08:28.009 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:28.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-30T11:08:28.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-03-30T11:08:28.010 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 150323855362 2024-03-30T11:08:28.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:28.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-30T11:08:28.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:29.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:08:29.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:29.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-30T11:08:29.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:30.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:08:30.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:30.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-30T11:08:30.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:30.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-30T11:08:30.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:30.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:08:30.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-30T11:08:30.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:30.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-30T11:08:30.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-30T11:08:30.886 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509456 2024-03-30T11:08:30.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:08:31.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-30T11:08:31.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:08:31.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:31.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:08:31.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:08:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:08:31.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:08:31.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:31.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:32.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:08:32.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:08:32.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:08:32.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-03-30T11:08:32.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T11:08:32.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-30T11:08:32.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-30T11:08:32.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:08:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-03-30T11:08:32.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-03-30T11:08:32.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:08:32.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T11:08:32.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T11:08:32.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:08:32.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-03-30T11:08:32.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-03-30T11:08:32.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:32.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:32.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:32.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:33.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:33.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-03-30T11:08:33.101 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:08:33.101 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": true, 2024-03-30T11:08:33.101 INFO:tasks.workunit.client.0.smithi003.stdout: "must": false, 2024-03-30T11:08:33.101 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "2024-03-16T11:06:53.106253+0000" 2024-03-30T11:08:33.101 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:33.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:33.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:07:48.180571+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:33.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:34.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:34.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:34.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:34.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:34.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:34.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:34.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:34.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:07:48.180571+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:34.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:35.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:35.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:35.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:35.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:35.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:35.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:35.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:35.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:07:48.180571+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:35.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:36.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:36.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:36.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:36.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:36.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:36.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:36.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:37.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:07:48.180571+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:37.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:38.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:38.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:38.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:38.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:38.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:38.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:38.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:38.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-16T11:06:53.106253+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:38.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:39.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:39.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:39.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:39.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:39.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:39.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:39.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-16T11:06:53.106253+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:40.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:40.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:40.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:40.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:40.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:40.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:40.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:41.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-16T11:06:53.106253+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:41.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:08:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:08:42.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:08:42.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:08:42.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:08:42.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:08:42.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:08:42.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:08:35.897798+0000 '>' 2024-03-30T11:07:48.180571+0000 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:08:42.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:08:42.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:08:42.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:08:42.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:08:42.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:08:42.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:08:42.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:08:42.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:08:42.808 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:42.808 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:08:42.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:08:42.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:42.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:08:42.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018889 2024-03-30T11:08:42.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018889 2024-03-30T11:08:42.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889' 2024-03-30T11:08:42.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:42.914 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:08:43.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855366 2024-03-30T11:08:43.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855366 2024-03-30T11:08:43.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855366' 2024-03-30T11:08:43.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:43.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:08:43.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-30T11:08:43.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-30T11:08:43.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018889 1-150323855366 2-64424509460' 2024-03-30T11:08:43.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:43.124 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018889 2024-03-30T11:08:43.124 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:43.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:08:43.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018889 2024-03-30T11:08:43.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:43.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018889 2024-03-30T11:08:43.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018889' 2024-03-30T11:08:43.127 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 128849018889 2024-03-30T11:08:43.127 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:43.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018889 2024-03-30T11:08:43.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:44.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:08:44.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:44.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018889 2024-03-30T11:08:44.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:44.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855366 2024-03-30T11:08:44.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:44.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:08:44.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855366 2024-03-30T11:08:44.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:44.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855366 2024-03-30T11:08:44.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855366' 2024-03-30T11:08:44.704 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 150323855366 2024-03-30T11:08:44.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:44.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855366 -lt 150323855366 2024-03-30T11:08:44.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:44.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-30T11:08:44.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:44.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:08:44.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-30T11:08:44.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-30T11:08:44.991 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509460 2024-03-30T11:08:44.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-30T11:08:44.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:08:45.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-30T11:08:45.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:08:45.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:45.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:08:45.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:08:45.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:08:45.914 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:08:45.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:45.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:46.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:08:46.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:08:46.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:08:46.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-30T11:08:46.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:08:46.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:08:46.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:08:46.554 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:46.554 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:08:46.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:08:46.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:46.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:08:46.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018891 2024-03-30T11:08:46.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018891 2024-03-30T11:08:46.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018891' 2024-03-30T11:08:46.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:46.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:08:46.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855368 2024-03-30T11:08:46.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855368 2024-03-30T11:08:46.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018891 1-150323855368' 2024-03-30T11:08:46.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:46.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:08:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-30T11:08:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-30T11:08:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018891 1-150323855368 2-64424509461' 2024-03-30T11:08:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:46.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018891 2024-03-30T11:08:46.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:46.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:08:46.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018891 2024-03-30T11:08:46.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:46.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018891 2024-03-30T11:08:46.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018891' 2024-03-30T11:08:46.897 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 128849018891 2024-03-30T11:08:46.898 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:47.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018891 2024-03-30T11:08:47.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:08:48.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:48.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018891 -lt 128849018891 2024-03-30T11:08:48.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:48.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855368 2024-03-30T11:08:48.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:48.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:08:48.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855368 2024-03-30T11:08:48.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:48.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855368 2024-03-30T11:08:48.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855368' 2024-03-30T11:08:48.480 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 150323855368 2024-03-30T11:08:48.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:48.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855368 2024-03-30T11:08:48.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:48.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-30T11:08:48.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:48.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:08:48.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-30T11:08:48.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:48.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-30T11:08:48.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-30T11:08:48.764 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509461 2024-03-30T11:08:48.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:08:49.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-30T11:08:49.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:08:49.078 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-30T11:08:49.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-03-30T11:08:49.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-30T11:08:49.376 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:08:49.376 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-30T11:08:35.897877+0000 21'10 36:100 [1,0] 1 [1,0] 1 21'10 2024-03-30T11:08:35.897798+0000 21'10 2024-03-30T11:08:35.897798+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:08:35.897798+0000 10 0 2024-03-30T11:08:49.376 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:08:49.376 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:08:49.377 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:08:49.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-30T11:08:49.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-03-30T11:08:49.652 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:08:49.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-03-30T11:08:49.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:08:49.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-30T11:08:49.665 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-30T11:08:49.665 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:08:49.665 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T11:08:49.665 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T11:08:49.665 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:08:49.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:08:49.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-30T11:08:49.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:08:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:08:50.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:08:50.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:08:50.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:08:50.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:08:50.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:08:50.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:08:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-03-30T11:08:51.778 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:08:51.778 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:08:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:08:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:08:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:08:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:08:52.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:08:52.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:08:52.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:08:52.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:08:52.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:08:52.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:08:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:08:52.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:08:52.085 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:52.089+0000 7feec4415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:52.087 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:52.091+0000 7feec4415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:52.089 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:52.093+0000 7feec4415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:08:52.346 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:08:52.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:52.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:52.607 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:52.611+0000 7feec4415640 -1 Falling back to public interface 2024-03-30T11:08:52.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:08:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:53.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:54.928 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:08:54.932+0000 7feec4415640 -1 osd.0 36 log_to_monitors true 2024-03-30T11:08:54.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:54.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:54.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:08:54.939 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:08:54.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:54.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:55.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:08:56.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:08:56.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:08:56.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:08:56.335 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:08:56.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:08:56.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:08:56.628 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/917392815,v1:127.0.0.1:6803/917392815] [v2:127.0.0.1:6804/917392815,v1:127.0.0.1:6805/917392815] exists,up 9f2deed3-8219-4be2-9058-f2a179463e93 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:08:56.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:08:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:08:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:08:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:08:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:08:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:08:56.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:08:56.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:08:56.991 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:08:56.991 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:08:56.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:08:56.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:56.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:08:57.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-30T11:08:57.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-30T11:08:57.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-03-30T11:08:57.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:57.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:08:57.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855371 2024-03-30T11:08:57.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855371 2024-03-30T11:08:57.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855371' 2024-03-30T11:08:57.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:08:57.236 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:08:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-30T11:08:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-30T11:08:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855371 2-64424509464' 2024-03-30T11:08:57.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:57.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-03-30T11:08:57.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:57.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:08:57.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-03-30T11:08:57.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:57.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-30T11:08:57.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-03-30T11:08:57.344 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 171798691842 2024-03-30T11:08:57.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:57.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-03-30T11:08:57.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:08:58.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:08:58.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:08:58.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-30T11:08:58.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:58.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855371 2024-03-30T11:08:58.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:58.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:08:58.927 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855371 2024-03-30T11:08:58.927 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:58.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855371 2024-03-30T11:08:58.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855371' 2024-03-30T11:08:58.928 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 150323855371 2024-03-30T11:08:58.928 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:08:59.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855371 -lt 150323855371 2024-03-30T11:08:59.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:08:59.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-30T11:08:59.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:08:59.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:08:59.215 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-30T11:08:59.215 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:08:59.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-30T11:08:59.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-30T11:08:59.217 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509464 2024-03-30T11:08:59.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:08:59.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-30T11:08:59.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:08:59.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:08:59.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:08:59.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:09:00.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:09:00.129 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:09:00.129 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:09:00.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:09:00.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:09:00.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:09:00.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:09:00.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-03-30T11:09:00.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:09:00.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:09:00.513 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-30T11:09:00.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-30T11:09:00.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:09:00.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:09:00.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:09:00.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:09:00.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-03-30T11:09:00.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:09:00.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:09:00.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:09:01.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:09:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-03-30T11:09:02.475 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-03-30T11:09:02.475 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:104778fc:::obj2:head# 2024-03-30T11:09:02.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:09:02.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:09:02.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:09:02.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:09:02.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:09:02.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:09:02.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:09:02.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:09:02.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:09:02.997 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:09:02.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:09:02.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:09:02.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:09:02.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:09:03.024 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:03.029+0000 7f0946b11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:03.027 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:03.031+0000 7f0946b11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:03.028 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:03.032+0000 7f0946b11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:09:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:09:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:09:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:09:03.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:09:03.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:03.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:09:03.289 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:09:03.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:03.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:09:03.549 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:03.554+0000 7f0946b11640 -1 Falling back to public interface 2024-03-30T11:09:03.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:04.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:04.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:04.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:09:04.576 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:09:04.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:04.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:09:04.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:05.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:05.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:05.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:09:05.868 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:09:05.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:05.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:09:05.990 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:05.993+0000 7f0946b11640 -1 osd.1 41 log_to_monitors true 2024-03-30T11:09:06.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:07.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:07.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:07.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:09:07.170 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:09:07.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:07.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:09:07.461 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3379185558,v1:127.0.0.1:6811/3379185558] [v2:127.0.0.1:6812/3379185558,v1:127.0.0.1:6813/3379185558] exists,up 7f6819c6-b233-4910-abda-a0a5ff413b05 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:09:07.462 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:09:07.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:09:07.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:09:07.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:09:07.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:09:07.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:09:07.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:09:07.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:09:07.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:09:07.545 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:09:07.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:09:07.829 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:09:07.829 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:09:07.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:09:07.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:09:07.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:09:07.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-30T11:09:07.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-30T11:09:07.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-30T11:09:07.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:09:07.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:09:08.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-03-30T11:09:08.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-03-30T11:09:08.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-03-30T11:09:08.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:09:08.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:09:08.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-30T11:09:08.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-30T11:09:08.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509468' 2024-03-30T11:09:08.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:09:08.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-30T11:09:08.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:09:08.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:09:08.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-30T11:09:08.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:09:08.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-30T11:09:08.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-30T11:09:08.155 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 171798691845 2024-03-30T11:09:08.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:09:08.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-03-30T11:09:08.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:09:09.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:09:09.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:09:09.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-30T11:09:09.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:09:09.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-03-30T11:09:09.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:09:09.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:09:09.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-03-30T11:09:09.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:09:09.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-03-30T11:09:09.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-03-30T11:09:09.737 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 193273528322 2024-03-30T11:09:09.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:09:10.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-03-30T11:09:10.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:09:10.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-30T11:09:10.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:09:10.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:09:10.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-30T11:09:10.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:09:10.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-30T11:09:10.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-30T11:09:10.022 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509468 2024-03-30T11:09:10.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:09:10.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-03-30T11:09:10.302 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:09:10.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:09:10.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:09:10.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:09:10.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:09:10.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:09:10.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:09:10.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:09:11.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:09:11.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:09:11.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:09:11.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-03-30T11:09:11.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-30T11:09:11.330 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-30T11:09:11.331 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:11.331 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:11.331 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:11.331 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:11.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:11.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-30T11:09:11.875 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-30T11:09:11.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:11.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T11:09:11.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:11.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:11.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:12.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:08:35.897798+0000 '>' 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:12.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:09:13.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:09:13.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:09:13.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:09:13.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:13.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:13.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:13.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:13.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:08:35.897798+0000 '>' 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:13.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:09:14.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:09:14.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:09:14.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:09:14.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:14.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:14.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:14.461 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:14.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:08:35.897798+0000 '>' 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:14.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:09:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:09:15.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:09:15.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:09:15.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:15.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:15.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:15.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:16.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:08:35.897798+0000 '>' 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:16.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:09:17.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:09:17.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:09:17.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:09:17.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:17.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:17.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:17.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:17.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:08:35.897798+0000 '>' 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:17.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:09:18.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:09:18.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:09:18.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:09:18.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:09:18.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:09:18.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:09:18.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:09:18.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:09:12.002498+0000 '>' 2024-03-30T11:08:35.897798+0000 2024-03-30T11:09:18.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:09:18.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-03-30T11:09:20.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-30T11:09:20.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:09:20.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:09:20.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:09:20.916 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:09:20.916 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:09:20.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:09:20.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:09:20.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:09:21.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-03-30T11:09:21.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-03-30T11:09:21.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849' 2024-03-30T11:09:21.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:09:21.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:09:21.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-03-30T11:09:21.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-03-30T11:09:21.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325' 2024-03-30T11:09:21.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:09:21.127 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:09:21.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-30T11:09:21.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-30T11:09:21.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325 2-64424509471' 2024-03-30T11:09:21.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:09:21.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691849 2024-03-30T11:09:21.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:09:21.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:09:21.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691849 2024-03-30T11:09:21.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:09:21.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-03-30T11:09:21.235 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 171798691849 2024-03-30T11:09:21.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-03-30T11:09:21.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:09:21.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691849 2024-03-30T11:09:21.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:09:22.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:09:22.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:09:22.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-03-30T11:09:22.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:09:22.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-03-30T11:09:22.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:09:22.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:09:22.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-03-30T11:09:22.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:09:22.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-03-30T11:09:22.821 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 193273528325 2024-03-30T11:09:22.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-03-30T11:09:22.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:09:23.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528326 -lt 193273528325 2024-03-30T11:09:23.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:09:23.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-30T11:09:23.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:09:23.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:09:23.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-30T11:09:23.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:09:23.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-30T11:09:23.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-30T11:09:23.105 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509471 2024-03-30T11:09:23.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:09:23.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-03-30T11:09:23.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-30T11:09:23.654 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:09:23.654 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-03-30T11:09:12.002621+0000 21'10 46:137 [1,0] 1 [1,0] 1 21'10 2024-03-30T11:09:12.002498+0000 21'10 2024-03-30T11:09:12.002498+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:09:12.002498+0000 9 0 2024-03-30T11:09:23.654 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:09:23.654 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:09:23.654 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:09:23.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-30T11:09:23.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-03-30T11:09:23.932 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:09:23.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.670+0000 7ff512e4c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:09:23.946 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:08:35.896+0000 7ff512e4c640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-30T11:09:23.947 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:09:12.000+0000 7f092aa8a640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-03-30T11:09:23.947 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:09:12.000+0000 7f092aa8a640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:09:23.947 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:09:12.000+0000 7f092aa8a640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-30T11:09:23.947 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:09:12.000+0000 7f092aa8a640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-03-30T11:09:23.947 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:09:12.000+0000 7f092aa8a640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:09:23.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:09:24.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:09:24.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:09:24.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:09:24.055 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:09:24.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:09:24.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:09:24.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:09:24.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:09:24.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:09:24.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:09:24.058 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:09:24.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:09:24.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:09:24.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:09:24.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:09:24.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:24.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:24.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:09:24.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:09:24.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:09:24.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:09:24.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:09:24.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:09:24.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:09:24.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:09:24.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:09:24.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:09:24.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:09:24.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:09:24.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:09:24.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:09:24.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:09:24.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:09:24.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:09:24.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:09:24.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:09:24.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:09:24.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:24.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:24.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:09:24.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:09:24.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:09:24.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:09:24.088 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:09:24.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:24.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:24.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-03-30T11:09:24.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:09:24.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:09:24.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:09:24.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:09:24.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:09:24.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:09:24.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:09:24.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:09:24.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:09:24.463 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:09:24.464 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:24.464 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:24.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:24.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:09:24.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:09:24.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:09:24.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:09:24.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:09:24.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:09:24.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:09:24.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:09:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:25.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:25.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:09:25.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:09:25.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:09:25.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-03-30T11:09:25.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-30T11:09:25.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:09:25.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:09:25.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:09:25.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:09:25.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:09:25.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:09:25.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:09:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d11f477-4cfc-46b9-a479-102c425fb74a 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 2d11f477-4cfc-46b9-a479-102c425fb74a 2024-03-30T11:09:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2d11f477-4cfc-46b9-a479-102c425fb74a' 2024-03-30T11:09:25.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:09:25.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDl8gdmvbGPKBAAEz/b/BSap7rFDxcPHnBZtA== 2024-03-30T11:09:25.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDl8gdmvbGPKBAAEz/b/BSap7rFDxcPHnBZtA=="}' 2024-03-30T11:09:25.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d11f477-4cfc-46b9-a479-102c425fb74a -i td/osd-scrub-repair/0/new.json 2024-03-30T11:09:25.880 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:09:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:09:25.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDl8gdmvbGPKBAAEz/b/BSap7rFDxcPHnBZtA== --osd-uuid 2d11f477-4cfc-46b9-a479-102c425fb74a 2024-03-30T11:09:25.921 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:25.925+0000 7fa54e68b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:25.931 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:25.935+0000 7fa54e68b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:25.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:25.936+0000 7fa54e68b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:25.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:25.937+0000 7fa54e68b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:09:25.933 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:25.937+0000 7fa54e68b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:09:25.933 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:25.937+0000 7fa54e68b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:09:26.023 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:26.027+0000 7fa54e68b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:09:31.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:09:31.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:09:31.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:09:31.574 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:09:31.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:09:31.952 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:09:31.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:09:31.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:09:31.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:09:31.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:09:31.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:09:31.980 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:31.984+0000 7fbf2ee37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:31.997 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:32.001+0000 7fbf2ee37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:31.998 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:32.002+0000 7fbf2ee37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:32.240 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:09:32.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:09:32.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:32.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:32.517 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:32.522+0000 7fbf2ee37640 -1 Falling back to public interface 2024-03-30T11:09:32.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:33.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:33.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:33.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:09:33.562 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:09:33.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:33.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:33.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:34.805 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:34.809+0000 7fbf2ee37640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:09:34.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:34.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:34.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:09:34.855 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:09:34.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:34.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:35.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:36.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:36.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:36.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:09:36.214 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:09:36.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:36.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:36.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:37.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:37.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:37.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:09:37.515 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:09:37.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:37.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:37.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:09:38.808 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:09:38.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:38.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:39.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:40.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:40.103 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:09:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:09:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:40.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:41.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:09:41.397 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:09:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:41.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:41.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:42.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:42.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:42.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:09:42.726 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:09:42.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:42.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:43.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:09:44.015 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:09:44.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:44.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:44.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:45.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:45.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:45.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:09:45.307 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:09:45.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:45.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:45.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:46.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:46.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:46.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:09:46.826 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:09:46.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:46.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:47.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:48.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:48.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:48.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:09:48.443 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:09:48.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:48.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:48.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:49.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:49.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:49.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:09:49.837 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:09:49.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:49.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:50.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:51.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:51.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:51.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:09:51.396 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:09:51.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:51.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:52.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:53.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:53.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:53.319 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:09:53.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:09:53.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:53.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:53.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:54.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:54.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:54.730 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:09:54.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:09:54.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:54.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:55.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:09:56.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:09:56.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:09:56.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:09:56.059 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:09:56.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:09:56.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:09:56.350 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1182907711,v1:127.0.0.1:6803/1182907711] [v2:127.0.0.1:6804/1182907711,v1:127.0.0.1:6805/1182907711] exists,up 2d11f477-4cfc-46b9-a479-102c425fb74a 2024-03-30T11:09:56.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:09:56.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:09:56.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:09:56.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:09:56.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a' 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a 2024-03-30T11:09:56.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:09:56.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAE8wdmUShrFhAAPmUAYr5k9eVnBmwUNWNIHQ== 2024-03-30T11:09:56.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAE8wdmUShrFhAAPmUAYr5k9eVnBmwUNWNIHQ=="}' 2024-03-30T11:09:56.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a -i td/osd-scrub-repair/1/new.json 2024-03-30T11:09:56.740 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:09:56.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:09:56.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAE8wdmUShrFhAAPmUAYr5k9eVnBmwUNWNIHQ== --osd-uuid 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a 2024-03-30T11:09:56.781 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.786+0000 7f3a8f824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:56.824 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.829+0000 7f3a8f824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:56.825 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.830+0000 7f3a8f824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:09:56.826 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.830+0000 7f3a8f824640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:09:56.826 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.830+0000 7f3a8f824640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:09:56.826 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.830+0000 7f3a8f824640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:09:56.891 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:09:56.895+0000 7f3a8f824640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:10:02.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:10:02.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:10:02.831 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:10:02.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:10:02.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:10:03.195 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:10:03.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:10:03.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:10:03.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:10:03.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:10:03.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:10:03.226 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:03.228+0000 7fab062d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:03.226 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:03.230+0000 7fab062d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:03.227 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:03.231+0000 7fab062d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:03.481 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:10:03.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:10:03.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:03.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:03.746 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:03.751+0000 7fab062d3640 -1 Falling back to public interface 2024-03-30T11:10:03.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:04.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:04.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:04.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:10:04.763 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:10:04.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:04.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:05.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:06.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:06.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:06.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:10:06.054 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:10:06.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:06.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:06.132 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:06.137+0000 7fab062d3640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:10:06.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:07.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:07.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:07.358 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:10:07.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:10:07.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:07.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:07.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:08.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:08.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:08.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:10:08.653 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:10:08.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:08.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:08.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:09.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:09.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:09.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:10:09.946 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:10:09.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:09.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:10.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:11.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:11.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:11.241 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:10:11.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:10:11.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:11.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:11.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:12.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:12.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:12.535 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:10:12.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:10:12.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:12.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:12.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:13.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:13.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:13.937 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:10:13.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:10:13.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:13.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:14.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:15.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:15.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:15.227 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:10:15.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:10:15.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:15.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:15.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:16.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:16.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:16.524 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:10:16.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:10:16.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:16.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:16.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:17.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:17.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:17.821 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:10:17.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:10:17.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:17.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:18.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:19.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:19.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:19.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:10:19.480 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:10:19.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:19.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:20.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:21.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:21.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:21.039 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:10:21.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:10:21.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:21.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:21.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:22.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:22.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:22.842 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:10:22.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:10:22.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:22.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:23.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:24.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:24.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:24.708 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:10:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:10:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:25.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:26.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:26.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:26.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:10:26.420 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:10:26.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:26.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:27.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:28.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:28.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:28.187 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:10:28.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:10:28.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:28.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:10:28.501 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2386359846,v1:127.0.0.1:6811/2386359846] [v2:127.0.0.1:6812/2386359846,v1:127.0.0.1:6813/2386359846] exists,up 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:10:28.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:10:28.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:10:28.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5052b121-c0fd-471d-9732-e44eae31e3ea 2024-03-30T11:10:28.509 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 5052b121-c0fd-471d-9732-e44eae31e3ea 2024-03-30T11:10:28.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5052b121-c0fd-471d-9732-e44eae31e3ea' 2024-03-30T11:10:28.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:10:28.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAk8wdmmjp+HxAArFpmMFWUA2hw79zyE6r4jA== 2024-03-30T11:10:28.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAk8wdmmjp+HxAArFpmMFWUA2hw79zyE6r4jA=="}' 2024-03-30T11:10:28.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5052b121-c0fd-471d-9732-e44eae31e3ea -i td/osd-scrub-repair/2/new.json 2024-03-30T11:10:28.875 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:10:28.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:10:28.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAk8wdmmjp+HxAArFpmMFWUA2hw79zyE6r4jA== --osd-uuid 5052b121-c0fd-471d-9732-e44eae31e3ea 2024-03-30T11:10:28.917 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:28.921+0000 7f62b16c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:28.951 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:28.956+0000 7f62b16c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:28.952 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:28.957+0000 7f62b16c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:28.952 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:28.957+0000 7f62b16c8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:10:28.952 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:28.957+0000 7f62b16c8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:10:28.953 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:28.957+0000 7f62b16c8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:10:29.027 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:29.032+0000 7f62b16c8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:10:34.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:10:34.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:10:34.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:10:34.764 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:10:34.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:10:35.134 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:10:35.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:10:35.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:10:35.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:10:35.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:10:35.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:10:35.163 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:35.167+0000 7fdc8ef49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:35.164 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:35.169+0000 7fdc8ef49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:35.165 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:35.170+0000 7fdc8ef49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:35.416 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:10:35.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:10:35.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:35.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:35.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:35.693 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:35.698+0000 7fdc8ef49640 -1 Falling back to public interface 2024-03-30T11:10:36.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:36.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:36.695 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:10:36.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:10:36.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:36.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:37.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:38.015 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:10:38.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:10:38.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:38.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:38.042 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:10:38.047+0000 7fdc8ef49640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:10:38.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:39.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:39.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:39.351 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:10:39.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:10:39.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:39.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:39.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:40.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:40.643 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:10:40.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:40.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:10:40.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:40.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:41.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:10:42.051 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:10:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:42.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:42.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:43.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:43.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:43.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:10:43.346 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:10:43.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:43.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:43.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:44.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:44.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:44.640 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:10:44.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:10:44.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:44.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:45.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:46.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:46.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:46.024 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:10:46.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:10:46.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:46.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:46.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:47.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:47.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:47.319 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:10:47.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:10:47.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:47.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:47.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:48.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:48.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:48.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:10:48.676 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:10:48.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:48.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:49.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:50.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:50.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:50.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:10:50.124 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:10:50.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:50.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:51.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:51.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:51.750 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:10:51.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:10:51.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:51.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:52.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:53.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:53.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:53.127 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:10:53.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:10:53.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:53.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:53.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:54.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:54.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:54.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:10:54.830 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:10:54.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:54.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:56.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:56.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:56.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:10:56.267 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:10:56.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:56.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:57.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:58.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:58.019 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:10:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:10:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:10:59.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:10:59.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:10:59.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:10:59.671 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:10:59.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:10:59.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:11:00.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:01.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:01.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:01.049 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:11:01.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:11:01.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:01.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:11:01.377 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2796525123,v1:127.0.0.1:6819/2796525123] [v2:127.0.0.1:6820/2796525123,v1:127.0.0.1:6821/2796525123] exists,up 5052b121-c0fd-471d-9732-e44eae31e3ea 2024-03-30T11:11:01.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:11:01.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:11:01.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:11:01.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-03-30T11:11:01.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-30T11:11:01.783 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T11:11:01.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:11:02.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-03-30T11:11:03.330 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 2 2024-03-30T11:11:03.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-30T11:11:03.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:11:03.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:11:03.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:11:03.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:11:03.343 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:11:03.343 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:11:03.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:11:03.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:11:03.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:11:03.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:11:03.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:11:03.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:11:03.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:11:03.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:11:03.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:11:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:11:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:11:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:11:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:03.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:11:03.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-30T11:11:03.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-30T11:11:03.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-30T11:11:03.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:03.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:11:04.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-30T11:11:04.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-30T11:11:04.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672974' 2024-03-30T11:11:04.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:04.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:11:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-30T11:11:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-30T11:11:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672974 2-64424509444' 2024-03-30T11:11:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:04.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-30T11:11:04.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:04.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:11:04.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-30T11:11:04.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:04.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-30T11:11:04.217 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836497 2024-03-30T11:11:04.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-30T11:11:04.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:04.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-30T11:11:04.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:11:05.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:11:05.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:05.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-30T11:11:05.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:11:06.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:11:06.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:07.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-30T11:11:07.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:07.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-30T11:11:07.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:07.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:11:07.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-30T11:11:07.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:07.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-30T11:11:07.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-30T11:11:07.158 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672974 2024-03-30T11:11:07.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:11:07.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-30T11:11:07.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:07.440 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-30T11:11:07.440 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:07.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:11:07.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-30T11:11:07.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:07.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-30T11:11:07.443 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509444 2024-03-30T11:11:07.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-30T11:11:07.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:11:07.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509444 2024-03-30T11:11:07.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:11:07.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:07.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:08.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:11:08.393 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:08.393 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:08.393 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:08.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:11:08.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-30T11:11:08.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:11:08.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-30T11:11:08.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:11:08.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:08.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:09.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:11:09.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:09.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:09.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:09.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:11:09.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:11:09.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:11:09.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:11:09.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:11:09.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:11:09.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:11:10.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:11:10.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:10.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:10.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:10.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:10.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:10.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:10.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:10.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:11:10.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:10.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:10.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:10.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:11:10.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:11:10.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:11:10.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-03-30T11:11:10.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-03-30T11:11:10.704 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-03-30T11:11:10.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:10.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:10.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:10.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:10.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:10.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-30T11:11:11.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:11:11.896 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:11:12.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:11:12.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:11:12.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:11:12.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-30T11:11:12.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:11:12.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:11:12.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:11:12.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-30T11:11:12.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:11:12.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:11:12.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:11:12.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:11:12.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:11:12.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-30T11:11:14.120 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-30T11:11:14.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:11:14.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:11:14.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:11:14.641 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:11:14.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:14.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:11:14.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:11:14.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:11:14.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:11:14.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:11:14.670 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:14.675+0000 7fd7649bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:14.673 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:14.678+0000 7fd7649bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:14.674 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:14.679+0000 7fd7649bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:14.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:11:14.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:14.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:15.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:15.193+0000 7fd7649bb640 -1 Falling back to public interface 2024-03-30T11:11:15.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:16.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:16.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:16.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:11:16.234 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:16.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:16.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:16.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:17.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:17.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:17.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:11:17.529 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:11:17.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:17.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:17.542 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:17.546+0000 7fd7649bb640 -1 osd.0 21 log_to_monitors true 2024-03-30T11:11:17.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:18.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:18.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:18.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:11:18.855 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:11:18.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:18.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:19.146 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/4093148944,v1:127.0.0.1:6803/4093148944] [v2:127.0.0.1:6804/4093148944,v1:127.0.0.1:6805/4093148944] exists,up 2d11f477-4cfc-46b9-a479-102c425fb74a 2024-03-30T11:11:19.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:11:19.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:11:19.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:11:19.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:11:19.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:11:19.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:11:19.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:11:19.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:11:19.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:11:19.508 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:19.508 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:11:19.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:11:19.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:19.508 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:11:19.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-30T11:11:19.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-30T11:11:19.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-30T11:11:19.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:19.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:11:19.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-30T11:11:19.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-30T11:11:19.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-03-30T11:11:19.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:19.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:11:19.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-30T11:11:19.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-30T11:11:19.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509448' 2024-03-30T11:11:19.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:19.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-30T11:11:19.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:19.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:11:19.825 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-30T11:11:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:19.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-30T11:11:19.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-30T11:11:19.827 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 107374182402 2024-03-30T11:11:19.827 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:20.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-30T11:11:20.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:20.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-30T11:11:20.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:20.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:11:20.124 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-30T11:11:20.124 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:20.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-30T11:11:20.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-30T11:11:20.126 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672978 2024-03-30T11:11:20.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:11:20.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-30T11:11:20.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:20.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-30T11:11:20.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:20.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:11:20.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-30T11:11:20.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:20.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-30T11:11:20.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-30T11:11:20.409 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509448 2024-03-30T11:11:20.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:11:20.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509448 2024-03-30T11:11:20.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:11:21.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:11:21.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:11:21.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509448 2024-03-30T11:11:21.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:11:21.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:21.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:22.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:22.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:11:22.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:22.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:22.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:22.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:11:22.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:11:22.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:11:22.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-30T11:11:22.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:11:22.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:11:22.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:11:22.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:11:23.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:11:23.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:11:23.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:11:23.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-30T11:11:23.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:11:23.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:11:23.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:11:23.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:11:23.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:11:23.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-03-30T11:11:25.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:25.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:11:25.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:11:25.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:11:25.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:11:25.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:11:25.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:11:25.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:11:25.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:11:25.224 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:11:25.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:25.225 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:11:25.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:11:25.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:11:25.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:11:25.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:11:25.253 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:25.258+0000 7f489920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:25.255 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:25.260+0000 7f489920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:25.256 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:25.261+0000 7f489920d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:25.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:11:25.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:11:25.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:25.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:25.771 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:25.776+0000 7f489920d640 -1 Falling back to public interface 2024-03-30T11:11:25.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:26.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:26.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:26.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:11:26.802 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:26.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:26.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:28.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:28.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:28.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:11:28.096 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:11:28.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:28.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:28.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:28.272+0000 7f489920d640 -1 osd.1 26 log_to_monitors true 2024-03-30T11:11:28.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:11:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:11:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:29.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/3550041135,v1:127.0.0.1:6811/3550041135] [v2:127.0.0.1:6812/3550041135,v1:127.0.0.1:6813/3550041135] exists,up 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:11:29.766 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:11:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:11:29.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:11:29.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:11:29.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:11:29.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:11:29.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:11:29.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:11:29.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:11:29.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:11:29.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:11:30.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:11:30.130 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:30.130 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:11:30.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:11:30.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:30.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:11:30.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-30T11:11:30.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-30T11:11:30.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-30T11:11:30.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:30.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:11:30.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-30T11:11:30.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-30T11:11:30.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-03-30T11:11:30.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:30.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:11:30.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-30T11:11:30.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-30T11:11:30.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509451' 2024-03-30T11:11:30.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:30.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-30T11:11:30.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:30.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:11:30.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-30T11:11:30.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:30.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-30T11:11:30.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-30T11:11:30.457 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 107374182405 2024-03-30T11:11:30.457 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:30.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-03-30T11:11:30.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:11:31.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:11:31.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:32.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-30T11:11:32.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:32.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-30T11:11:32.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:32.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:11:32.058 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-30T11:11:32.058 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:32.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-30T11:11:32.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-30T11:11:32.060 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 128849018882 2024-03-30T11:11:32.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:11:32.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-30T11:11:32.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:32.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-30T11:11:32.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:32.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:11:32.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-30T11:11:32.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:32.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-30T11:11:32.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-30T11:11:32.343 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509451 2024-03-30T11:11:32.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:11:32.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509451 2024-03-30T11:11:32.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:11:32.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:32.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:32.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:11:32.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:32.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:32.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:32.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:32.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:32.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:32.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:33.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:11:33.257 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:33.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:33.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:33.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:11:33.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:11:33.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:11:33.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:11:33.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:11:33.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-30T11:11:34.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:11:34.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:11:34.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-30T11:11:35.707 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:104778fc:::obj2:head# 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:11:36.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:36.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:11:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:11:36.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:11:36.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:11:36.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:36.264+0000 7f73a72d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:36.262 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:36.266+0000 7f73a72d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:36.264 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:36.268+0000 7f73a72d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:36.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:11:36.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:11:36.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:11:36.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:11:36.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:11:36.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:36.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:11:36.522 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:11:36.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:36.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:36.781 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:36.785+0000 7f73a72d5640 -1 Falling back to public interface 2024-03-30T11:11:37.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:37.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:37.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:37.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:11:37.806 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:37.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:37.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:38.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:39.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:39.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:39.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:11:39.100 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:11:39.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:39.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:39.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:39.192+0000 7f73a72d5640 -1 osd.0 31 log_to_monitors true 2024-03-30T11:11:39.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:40.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:40.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:40.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:11:40.443 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:11:40.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:40.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:11:40.741 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/450070654,v1:127.0.0.1:6803/450070654] [v2:127.0.0.1:6804/450070654,v1:127.0.0.1:6805/450070654] exists,up 2d11f477-4cfc-46b9-a479-102c425fb74a 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:11:40.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:11:40.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:11:40.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:11:40.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:11:40.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:11:40.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:11:40.825 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:11:41.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:11:41.108 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:41.108 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:11:41.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:11:41.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:41.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-30T11:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-30T11:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-30T11:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:41.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:11:41.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-30T11:11:41.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-30T11:11:41.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-03-30T11:11:41.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:41.318 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-30T11:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-30T11:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509455' 2024-03-30T11:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-30T11:11:41.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:41.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:11:41.424 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-30T11:11:41.424 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:41.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-30T11:11:41.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-30T11:11:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 150323855362 2024-03-30T11:11:41.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:41.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-30T11:11:41.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:11:42.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:11:42.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:43.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-30T11:11:43.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:43.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018885 2024-03-30T11:11:43.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:43.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:11:43.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018885 2024-03-30T11:11:43.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:43.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-30T11:11:43.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-03-30T11:11:43.022 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 128849018885 2024-03-30T11:11:43.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:11:43.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-30T11:11:43.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:43.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-30T11:11:43.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:43.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:11:43.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-30T11:11:43.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:43.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-30T11:11:43.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-30T11:11:43.312 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509455 2024-03-30T11:11:43.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:11:43.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-30T11:11:43.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:11:43.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:43.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:43.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:43.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:44.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:11:44.241 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:44.242 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:44.242 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:44.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:11:44.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:11:44.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:11:44.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-30T11:11:44.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:11:44.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:11:44.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:11:44.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:11:44.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-30T11:11:44.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:11:44.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:11:44.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:11:44.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:11:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:11:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-30T11:11:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:11:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:11:45.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:11:45.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:11:45.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:11:45.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-30T11:11:46.374 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-03-30T11:11:46.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:46.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:11:46.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:11:46.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:11:46.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:11:46.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:11:46.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:11:46.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:11:46.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:11:46.895 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:11:46.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:11:46.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:11:46.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:11:46.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:11:46.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:11:46.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:11:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:46.928+0000 7f78978b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:46.930+0000 7f78978b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:46.927 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:46.932+0000 7f78978b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:11:47.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:11:47.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:11:47.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:47.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:47.440 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:47.445+0000 7f78978b5640 -1 Falling back to public interface 2024-03-30T11:11:47.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:48.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:48.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:48.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:11:48.467 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:48.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:48.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:48.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:11:49.697 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:11:49.701+0000 7f78978b5640 -1 osd.1 36 log_to_monitors true 2024-03-30T11:11:49.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:11:49.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:11:49.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:11:49.764 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:11:49.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:11:49.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 40 up_thru 35 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/2306785466,v1:127.0.0.1:6811/2306785466] [v2:127.0.0.1:6812/2306785466,v1:127.0.0.1:6813/2306785466] exists,up 95a1c4f1-7fab-4a97-b1bc-c0f8c9eb745a 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:11:50.094 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:11:50.095 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:11:50.095 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:11:50.095 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:11:50.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:11:50.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:11:50.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:11:50.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:11:50.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:11:50.178 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:11:50.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:11:50.490 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:11:50.490 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:11:50.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:11:50.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:50.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:11:50.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-30T11:11:50.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-30T11:11:50.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365' 2024-03-30T11:11:50.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:50.603 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:11:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-30T11:11:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-30T11:11:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-03-30T11:11:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:11:50.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:11:50.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-30T11:11:50.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-30T11:11:50.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509458' 2024-03-30T11:11:50.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:50.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855365 2024-03-30T11:11:50.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:50.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:11:50.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855365 2024-03-30T11:11:50.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:50.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-30T11:11:50.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-03-30T11:11:50.859 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 150323855365 2024-03-30T11:11:50.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:51.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855364 -lt 150323855365 2024-03-30T11:11:51.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:11:52.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:11:52.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:11:52.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-30T11:11:52.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:52.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-30T11:11:52.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:52.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:11:52.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-30T11:11:52.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:52.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-30T11:11:52.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-30T11:11:52.440 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 171798691842 2024-03-30T11:11:52.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:11:52.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-30T11:11:52.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:11:52.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-30T11:11:52.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:11:52.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:11:52.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-30T11:11:52.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:11:52.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-30T11:11:52.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-30T11:11:52.723 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509458 2024-03-30T11:11:52.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:11:53.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-30T11:11:53.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:11:53.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:53.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:11:53.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:11:53.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:11:53.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:11:53.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:11:53.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:11:53.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:11:53.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:11:53.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:11:53.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-30T11:11:54.093 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0: { 2024-03-30T11:11:54.093 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-30T11:11:54.093 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:11:54.108 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1: { 2024-03-30T11:11:54.108 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-30T11:11:54.108 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:11:54.122 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2: { 2024-03-30T11:11:54.122 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-30T11:11:54.122 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:11:54.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-03-30T11:11:54.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T11:11:54.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-30T11:11:54.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-30T11:11:54.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:11:54.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-03-30T11:11:54.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-03-30T11:11:54.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:11:54.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T11:11:54.416 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T11:11:54.416 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:11:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-03-30T11:11:54.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-03-30T11:11:54.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:11:54.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:11:54.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:11:54.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:11:55.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:55.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-03-30T11:11:55.116 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:11:55.116 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": true, 2024-03-30T11:11:55.116 INFO:tasks.workunit.client.0.smithi003.stdout: "must": false, 2024-03-30T11:11:55.116 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "2024-03-16T11:10:15.121628+0000" 2024-03-30T11:11:55.116 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:11:55.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:11:55.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:11:55.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:11:55.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:11:55.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:11:01.752375+0000 '>' 2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:55.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:11:56.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:11:56.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:11:56.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:11:56.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:11:56.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:11:56.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:11:56.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:11:56.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:11:01.752375+0000 '>' 2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:56.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:11:57.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:11:57.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:11:57.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:11:57.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:11:57.694 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:11:57.694 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:11:57.694 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:11:58.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:11:01.752375+0000 '>' 2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:58.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:11:59.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:11:59.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:11:59.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:11:59.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:11:59.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:11:59.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:11:59.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:11:59.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:11:01.752375+0000 '>' 2024-03-30T11:11:01.752375+0000 2024-03-30T11:11:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:12:00.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:12:00.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:12:00.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:12:00.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:12:00.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:12:00.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:12:00.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:12:00.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:11:01.752375+0000 '>' 2024-03-30T11:11:01.752375+0000 2024-03-30T11:12:00.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:12:01.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:12:01.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:12:01.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:12:01.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:12:01.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:12:01.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:12:01.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:11:55.766128+0000 '>' 2024-03-30T11:11:01.752375+0000 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:12:01.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:12:02.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:12:02.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:12:02.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:12:02.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:12:02.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:12:02.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:12:02.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:12:02.283 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:12:02.283 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:12:02.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:12:02.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:12:02.283 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:12:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855368 2024-03-30T11:12:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855368 2024-03-30T11:12:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368' 2024-03-30T11:12:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:12:02.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:12:02.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-30T11:12:02.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-30T11:12:02.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-03-30T11:12:02.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:12:02.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:12:02.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-30T11:12:02.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-30T11:12:02.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509461' 2024-03-30T11:12:02.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:12:02.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855368 2024-03-30T11:12:02.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:12:02.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:12:02.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855368 2024-03-30T11:12:02.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:12:02.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855368 2024-03-30T11:12:02.601 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 150323855368 2024-03-30T11:12:02.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-03-30T11:12:02.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:12:02.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855368 2024-03-30T11:12:02.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:12:03.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:12:03.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:12:04.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855368 -lt 150323855368 2024-03-30T11:12:04.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:12:04.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-03-30T11:12:04.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:12:04.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:12:04.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-03-30T11:12:04.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:12:04.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-30T11:12:04.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-03-30T11:12:04.178 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 171798691845 2024-03-30T11:12:04.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:12:04.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-30T11:12:04.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:12:04.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-30T11:12:04.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:12:04.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:12:04.461 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-30T11:12:04.461 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:12:04.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-30T11:12:04.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-30T11:12:04.463 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509461 2024-03-30T11:12:04.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:12:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-30T11:12:04.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:12:04.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:12:04.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:12:05.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:12:05.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:12:05.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:12:05.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:12:05.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:12:05.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:12:05.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:12:05.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-03-30T11:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:12:06.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:12:06.061 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:12:06.061 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:12:06.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:12:06.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:12:06.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:12:06.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-03-30T11:12:06.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-03-30T11:12:06.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370' 2024-03-30T11:12:06.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:12:06.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:12:06.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691847 2024-03-30T11:12:06.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691847 2024-03-30T11:12:06.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-03-30T11:12:06.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:12:06.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:12:06.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-30T11:12:06.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-30T11:12:06.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509463' 2024-03-30T11:12:06.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:12:06.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855370 2024-03-30T11:12:06.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:12:06.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:12:06.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855370 2024-03-30T11:12:06.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:12:06.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-03-30T11:12:06.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-03-30T11:12:06.379 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 150323855370 2024-03-30T11:12:06.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:12:06.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855369 -lt 150323855370 2024-03-30T11:12:06.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:12:07.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:12:07.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:12:07.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-03-30T11:12:07.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:12:07.954 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691847 2024-03-30T11:12:07.954 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:12:07.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:12:07.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691847 2024-03-30T11:12:07.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:12:07.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691847 2024-03-30T11:12:07.956 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 171798691847 2024-03-30T11:12:07.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-03-30T11:12:07.956 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:12:08.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691847 2024-03-30T11:12:08.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:12:08.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-30T11:12:08.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:12:08.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:12:08.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-30T11:12:08.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:12:08.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-30T11:12:08.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-30T11:12:08.278 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509463 2024-03-30T11:12:08.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:12:08.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-30T11:12:08.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-03-30T11:12:08.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-30T11:12:08.828 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:12:08.828 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-30T11:11:55.766226+0000 21'10 41:97 [1,0] 1 [1,0] 1 21'10 2024-03-30T11:11:55.766128+0000 21'10 2024-03-30T11:11:55.766128+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:11:55.766128+0000 10 0 2024-03-30T11:12:08.828 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:12:08.828 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:12:08.828 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:12:08.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-30T11:12:08.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-03-30T11:12:09.102 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:12:09.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:12:09.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:12:09.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:12:09.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:12:09.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:12:09.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:12:09.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:12:09.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:12:09.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:12:09.224 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:12:09.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:12:09.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:12:09.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:12:09.225 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:12:09.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:12:09.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:12:09.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:12:09.242 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:12:09.242 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:09.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:09.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:12:09.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:12:09.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:12:09.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:12:09.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:12:09.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:12:09.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:12:09.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:12:09.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:12:09.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:12:09.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:12:09.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:12:09.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:12:09.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:12:09.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:12:09.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:12:09.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:12:09.250 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:12:09.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:12:09.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:12:09.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:12:09.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:12:09.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:09.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:09.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:12:09.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:12:09.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:12:09.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:12:09.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:12:09.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:09.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:09.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:12:09.257 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:12:09.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:12:09.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:12:09.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:12:09.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:12:09.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:12:09.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:12:09.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:12:09.621 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:12:09.621 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:09.621 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:09.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:12:09.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:12:09.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:12:09.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:12:09.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:12:09.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:12:09.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:12:09.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:12:09.799 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:12:09.799 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:12:09.799 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:12:09.800 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:12:09.800 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:09.800 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:09.800 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:12:09.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:12:09.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:12:09.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:12:09.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:12:09.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:12:09.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:12:09.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:12:09.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:12:09.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-03-30T11:12:09.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:12:09.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:12:09.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:12:09.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:12:09.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:12:09.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:12:10.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:12:10.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:12:10.784 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:12:10.784 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:12:10.784 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:10.784 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:10.784 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:12:10.784 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:12:10.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:12:10.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-30T11:12:10.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:12:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:12:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:12:10.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1c8fb56-a78d-4e49-98b7-eba6cd492902 2024-03-30T11:12:10.822 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 a1c8fb56-a78d-4e49-98b7-eba6cd492902 2024-03-30T11:12:10.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a1c8fb56-a78d-4e49-98b7-eba6cd492902' 2024-03-30T11:12:10.822 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:12:10.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCK8wdmXJAwMhAA1wpAj8S9qXC0xmqCinXtiw== 2024-03-30T11:12:10.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCK8wdmXJAwMhAA1wpAj8S9qXC0xmqCinXtiw=="}' 2024-03-30T11:12:10.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1c8fb56-a78d-4e49-98b7-eba6cd492902 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:12:11.046 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:12:11.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:12:11.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCK8wdmXJAwMhAA1wpAj8S9qXC0xmqCinXtiw== --osd-uuid a1c8fb56-a78d-4e49-98b7-eba6cd492902 2024-03-30T11:12:11.087 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.092+0000 7f9691747640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:11.089 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.094+0000 7f9691747640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:11.090 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.095+0000 7f9691747640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:11.090 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.095+0000 7f9691747640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:12:11.090 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.095+0000 7f9691747640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:12:11.091 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.095+0000 7f9691747640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:12:11.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:11.193+0000 7f9691747640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:12:16.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:12:16.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:12:16.771 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:12:16.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:12:16.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:12:17.142 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:12:17.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:12:17.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:12:17.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:12:17.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:12:17.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:12:17.171 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:17.176+0000 7f7c9c804640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:17.186 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:17.191+0000 7f7c9c804640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:17.187 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:17.192+0000 7f7c9c804640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:17.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:12:17.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:12:17.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:17.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:17.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:17.704 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:17.710+0000 7f7c9c804640 -1 Falling back to public interface 2024-03-30T11:12:18.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:18.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:18.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:12:18.706 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:12:18.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:18.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:18.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:19.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:19.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:19.996 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:12:19.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:12:19.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:19.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:20.035 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:20.040+0000 7f7c9c804640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:12:20.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:21.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:21.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:21.303 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:12:21.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:12:21.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:21.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:21.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:22.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:22.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:22.600 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:12:22.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:12:22.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:22.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:22.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:23.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:23.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:23.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:12:23.957 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:12:23.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:23.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:24.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:25.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:25.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:25.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:12:25.251 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:12:25.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:25.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:25.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:26.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:26.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:26.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:12:26.548 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:12:26.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:26.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:26.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:27.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:27.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:27.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:12:27.974 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:12:27.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:27.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:28.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:29.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:29.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:29.266 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:12:29.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:12:29.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:29.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:29.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:30.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:30.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:30.588 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:12:30.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:12:30.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:30.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:30.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:31.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:31.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:31.958 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:12:31.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:12:31.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:31.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:32.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:33.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:33.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:33.596 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:12:33.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:12:33.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:33.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:33.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:34.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:34.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:34.973 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:12:34.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:12:34.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:34.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:35.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:36.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:36.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:36.692 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:12:36.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:12:36.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:36.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:37.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:38.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:38.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:38.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:12:38.640 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:12:38.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:38.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:39.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:40.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:40.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:40.122 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:12:40.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:12:40.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:40.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:40.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:41.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:41.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:41.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:12:41.752 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:12:41.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:41.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:12:42.061 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2418211056,v1:127.0.0.1:6803/2418211056] [v2:127.0.0.1:6804/2418211056,v1:127.0.0.1:6805/2418211056] exists,up a1c8fb56-a78d-4e49-98b7-eba6cd492902 2024-03-30T11:12:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:12:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:12:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:12:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-30T11:12:42.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:12:42.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:12:42.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2a619de-eed4-45b0-b913-c1cbee26499b 2024-03-30T11:12:42.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a2a619de-eed4-45b0-b913-c1cbee26499b' 2024-03-30T11:12:42.069 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 a2a619de-eed4-45b0-b913-c1cbee26499b 2024-03-30T11:12:42.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:12:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCq8wdmuBcuBRAA7UlmjMMwlLc1I1NDPlY37A== 2024-03-30T11:12:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCq8wdmuBcuBRAA7UlmjMMwlLc1I1NDPlY37A=="}' 2024-03-30T11:12:42.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2a619de-eed4-45b0-b913-c1cbee26499b -i td/osd-scrub-repair/1/new.json 2024-03-30T11:12:42.435 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:12:42.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:12:42.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCq8wdmuBcuBRAA7UlmjMMwlLc1I1NDPlY37A== --osd-uuid a2a619de-eed4-45b0-b913-c1cbee26499b 2024-03-30T11:12:42.477 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.482+0000 7faa5380e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:42.511 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.516+0000 7faa5380e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:42.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.517+0000 7faa5380e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:42.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.518+0000 7faa5380e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:12:42.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.518+0000 7faa5380e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:12:42.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.518+0000 7faa5380e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:12:42.578 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:42.583+0000 7faa5380e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:12:48.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:12:48.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:12:48.628 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:12:48.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:12:48.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:12:48.997 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:12:48.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:12:48.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:12:48.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:12:48.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:12:48.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:12:49.026 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:49.030+0000 7f2195773640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:49.033 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:49.037+0000 7f2195773640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:49.034 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:49.038+0000 7f2195773640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:49.288 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:12:49.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:12:49.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:49.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:49.569 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:49.573+0000 7f2195773640 -1 Falling back to public interface 2024-03-30T11:12:49.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:50.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:50.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:50.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:12:50.571 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:12:50.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:50.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:50.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:51.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:51.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:51.864 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:12:51.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:12:51.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:51.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:51.958 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:12:51.962+0000 7f2195773640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:12:52.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:53.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:53.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:53.218 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:12:53.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:12:53.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:53.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:53.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:54.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:54.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:54.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:12:54.510 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:12:54.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:54.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:54.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:55.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:55.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:55.806 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:12:55.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:12:55.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:55.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:56.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:57.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:57.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:57.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:12:57.136 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:12:57.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:57.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:57.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:58.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:58.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:58.430 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:12:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:12:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:12:58.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:12:59.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:12:59.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:12:59.806 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:12:59.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:12:59.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:12:59.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:00.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:01.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:01.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:01.127 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:13:01.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:13:01.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:01.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:01.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:02.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:02.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:02.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:13:02.418 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:13:02.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:02.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:02.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:03.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:03.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:03.814 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:13:03.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:13:03.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:03.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:05.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:05.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:05.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:13:05.433 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:13:05.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:05.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:06.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:07.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:07.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:07.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:13:07.238 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:13:07.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:07.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:07.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:08.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:08.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:08.811 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:13:08.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:13:08.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:08.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:09.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:10.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:10.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:10.368 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:13:10.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:13:10.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:10.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:10.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:11.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:11.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:11.923 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:13:11.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:13:11.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:11.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:12.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:13.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:13.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:13.574 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:13:13.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:13:13.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:13.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:14.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:15.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:15.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:15.024 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:13:15.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:13:15.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:15.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:13:15.343 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2119386172,v1:127.0.0.1:6811/2119386172] [v2:127.0.0.1:6812/2119386172,v1:127.0.0.1:6813/2119386172] exists,up a2a619de-eed4-45b0-b913-c1cbee26499b 2024-03-30T11:13:15.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:13:15.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:13:15.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:13:15.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:13:15.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:13:15.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=71a627bd-351c-46a6-b45b-c6a7a655aa44 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 71a627bd-351c-46a6-b45b-c6a7a655aa44' 2024-03-30T11:13:15.350 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 71a627bd-351c-46a6-b45b-c6a7a655aa44 2024-03-30T11:13:15.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:13:15.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDL8wdmOyMOFhAABDiif1rH5hl9/qGFkM7rlw== 2024-03-30T11:13:15.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDL8wdmOyMOFhAABDiif1rH5hl9/qGFkM7rlw=="}' 2024-03-30T11:13:15.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 71a627bd-351c-46a6-b45b-c6a7a655aa44 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:13:15.734 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:13:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:13:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDL8wdmOyMOFhAABDiif1rH5hl9/qGFkM7rlw== --osd-uuid 71a627bd-351c-46a6-b45b-c6a7a655aa44 2024-03-30T11:13:15.777 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.782+0000 7f5ca7411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:15.819 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.824+0000 7f5ca7411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:15.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.825+0000 7f5ca7411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:15.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.825+0000 7f5ca7411640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:13:15.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.825+0000 7f5ca7411640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:13:15.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.825+0000 7f5ca7411640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:13:15.894 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:15.899+0000 7f5ca7411640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:13:21.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:13:21.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:13:21.643 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:13:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:13:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:13:22.010 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:13:22.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:13:22.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:13:22.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:13:22.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:13:22.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:13:22.038 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:22.044+0000 7f3c986c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:22.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:22.063+0000 7f3c986c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:22.059 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:22.064+0000 7f3c986c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:22.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:13:22.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:13:22.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:13:22.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:13:22.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:13:22.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:22.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:13:22.330 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:13:22.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:22.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:22.589 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:22.594+0000 7f3c986c8640 -1 Falling back to public interface 2024-03-30T11:13:22.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:23.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:23.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:23.617 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:13:23.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:13:23.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:23.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:23.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:24.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:24.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:24.913 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:13:24.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:13:24.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:24.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:24.950 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:24.956+0000 7f3c986c8640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:13:25.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:26.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:26.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:26.252 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:13:26.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:13:26.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:26.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:26.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:27.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:27.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:27.548 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:13:27.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:13:27.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:27.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:28.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:29.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:29.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:29.133 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:13:29.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:13:29.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:29.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:29.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:30.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:30.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:30.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:13:30.428 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:13:30.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:30.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:30.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:31.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:31.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:31.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:13:31.746 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:13:31.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:31.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:32.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:33.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:33.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:33.150 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:13:33.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:13:33.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:33.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:33.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:34.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:34.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:34.439 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:13:34.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:13:34.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:34.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:34.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:35.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:35.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:35.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:13:35.739 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:13:35.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:35.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:36.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:37.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:37.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:37.110 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:13:37.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:13:37.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:37.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:37.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:38.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:38.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:38.645 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:13:38.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:13:38.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:38.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:39.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:40.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:40.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:40.032 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:13:40.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:13:40.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:40.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:40.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:41.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:13:41.775 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:13:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:41.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:42.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:43.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:43.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:43.503 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:13:43.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:13:43.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:43.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:44.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:45.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:45.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:45.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:13:45.297 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:13:45.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:45.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:46.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:47.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:47.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:47.041 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:13:47.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:13:47.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:47.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:13:47.415 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2003728258,v1:127.0.0.1:6819/2003728258] [v2:127.0.0.1:6820/2003728258,v1:127.0.0.1:6821/2003728258] exists,up 71a627bd-351c-46a6-b45b-c6a7a655aa44 2024-03-30T11:13:47.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:13:47.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:13:47.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:13:47.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-03-30T11:13:47.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-30T11:13:47.793 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T11:13:47.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:13:48.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-03-30T11:13:49.259 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 2 2024-03-30T11:13:49.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-30T11:13:49.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:13:49.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:13:49.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:13:49.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:13:49.271 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:13:49.271 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:13:49.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:13:49.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:13:49.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:13:49.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:13:49.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:13:49.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:13:49.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:13:49.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:13:49.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:13:49.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:13:49.875 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:13:49.876 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:13:49.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:13:49.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:13:49.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:13:49.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-30T11:13:49.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-30T11:13:49.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-30T11:13:49.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:13:49.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:13:50.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-30T11:13:50.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-30T11:13:50.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672971' 2024-03-30T11:13:50.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:13:50.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:13:50.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-30T11:13:50.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-30T11:13:50.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672971 2-64424509444' 2024-03-30T11:13:50.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:13:50.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-30T11:13:50.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:13:50.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:13:50.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-30T11:13:50.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:13:50.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-30T11:13:50.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-30T11:13:50.213 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836495 2024-03-30T11:13:50.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:13:50.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-30T11:13:50.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:13:51.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:13:51.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:13:51.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-03-30T11:13:51.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:13:51.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-30T11:13:51.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:13:51.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:13:51.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-30T11:13:51.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:13:51.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-30T11:13:51.789 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672971 2024-03-30T11:13:51.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-30T11:13:51.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:13:52.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672971 2024-03-30T11:13:52.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:13:52.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-30T11:13:52.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:13:52.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:13:52.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-30T11:13:52.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:13:52.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-30T11:13:52.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-30T11:13:52.082 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509444 2024-03-30T11:13:52.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:13:52.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-30T11:13:52.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:13:52.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:13:52.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:13:52.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:13:52.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:13:52.813 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:13:52.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:13:52.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:13:52.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:13:52.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:13:52.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:13:53.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:13:53.095 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:13:53.095 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:13:53.095 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:13:53.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:13:53.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:13:53.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:13:53.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-03-30T11:13:53.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-03-30T11:13:53.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:13:53.631 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:13:53.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:13:53.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:13:53.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:13:54.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:13:54.220 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:13:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:13:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:13:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:13:54.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-30T11:13:55.771 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-30T11:13:56.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:13:56.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:13:56.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:13:56.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:13:56.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:13:56.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-30T11:13:56.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:13:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:13:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:13:56.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:13:56.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:13:56.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:13:56.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:13:56.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:13:56.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:13:56.323 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:56.328+0000 7ff977b93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:56.325 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:56.330+0000 7ff977b93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:56.332+0000 7ff977b93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:13:56.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:56.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:13:56.840 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:56.846+0000 7ff977b93640 -1 Falling back to public interface 2024-03-30T11:13:56.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:57.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:57.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:57.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:13:57.871 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:13:57.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:57.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:13:58.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:13:59.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:13:59.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:13:59.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:13:59.183 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:13:59.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:13:59.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:13:59.237 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:13:59.241+0000 7ff977b93640 -1 osd.0 21 log_to_monitors true 2024-03-30T11:13:59.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:00.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:00.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:00.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:14:00.475 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:14:00.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:00.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3926788931,v1:127.0.0.1:6803/3926788931] [v2:127.0.0.1:6804/3926788931,v1:127.0.0.1:6805/3926788931] exists,up a1c8fb56-a78d-4e49-98b7-eba6cd492902 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:14:00.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:14:00.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:14:00.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:14:00.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:14:00.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:14:00.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:14:00.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:14:00.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:14:00.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:14:01.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:14:01.131 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:14:01.131 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:14:01.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:14:01.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:01.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:14:01.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-30T11:14:01.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-30T11:14:01.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-30T11:14:01.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:01.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:14:01.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-30T11:14:01.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-30T11:14:01.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-03-30T11:14:01.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:01.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:14:01.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-30T11:14:01.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-30T11:14:01.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672975 2-64424509447' 2024-03-30T11:14:01.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:01.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-30T11:14:01.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:01.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:14:01.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-30T11:14:01.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:01.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-30T11:14:01.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-30T11:14:01.465 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 107374182402 2024-03-30T11:14:01.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:01.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-30T11:14:01.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:14:02.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:14:02.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:03.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-30T11:14:03.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:14:04.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:14:04.051 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:04.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-30T11:14:04.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:04.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-30T11:14:04.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:04.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:14:04.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-30T11:14:04.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-30T11:14:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-30T11:14:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672975 2024-03-30T11:14:04.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:14:04.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-30T11:14:04.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:04.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-30T11:14:04.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:04.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:14:04.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-30T11:14:04.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:04.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-30T11:14:04.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-30T11:14:04.635 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509447 2024-03-30T11:14:04.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:14:04.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-03-30T11:14:04.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:14:04.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:14:04.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:14:05.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:14:05.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:14:05.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:14:05.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:14:05.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:14:05.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:14:05.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:14:05.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:14:05.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:14:05.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:14:05.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:14:05.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:14:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-30T11:14:05.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:14:06.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-03-30T11:14:06.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-03-30T11:14:06.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:14:06.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:14:06.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:14:06.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:14:06.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-03-30T11:14:06.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-03-30T11:14:06.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:14:06.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:14:06.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:14:06.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:14:06.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:06.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-03-30T11:14:06.863 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:14:06.864 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": false, 2024-03-30T11:14:06.864 INFO:tasks.workunit.client.0.smithi003.stdout: "must": false, 2024-03-30T11:14:06.864 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "2024-03-23T11:12:26.869527+0000" 2024-03-30T11:14:06.864 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:14:06.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:06.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T11:14:06.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:06.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:14:06.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:14:07.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:13:47.747873+0000 '>' 2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:07.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:14:08.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:14:08.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:14:08.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:14:08.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:14:08.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:14:08.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:14:08.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:14:08.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:13:47.747873+0000 '>' 2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:08.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:14:09.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:14:09.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:14:09.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:14:09.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:14:09.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:14:09.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:14:09.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:14:09.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:13:47.747873+0000 '>' 2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:09.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:14:10.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:14:10.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:14:10.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T11:14:10.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:14:10.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:14:10.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:14:10.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:14:11.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:14:07.903042+0000 '>' 2024-03-30T11:13:47.747873+0000 2024-03-30T11:14:11.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:14:11.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-30T11:14:11.290 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:14:11.291 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-30T11:14:08.136868+0000 21'1 26:66 [1,0] 1 [1,0] 1 21'1 2024-03-30T11:14:07.903042+0000 21'1 2024-03-30T11:14:07.903042+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:14:07.903042+0000 1 0 2024-03-30T11:14:11.291 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:14:11.291 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:14:11.291 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:14:11.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-30T11:14:13.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-30T11:14:13.582 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:14:13.582 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-30T11:14:08.136868+0000 21'1 26:66 [1,0] 1 [1,0] 1 21'1 2024-03-30T11:14:07.903042+0000 21'1 2024-03-30T11:14:07.903042+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:14:07.903042+0000 1 0 2024-03-30T11:14:13.582 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:14:13.582 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:14:13.582 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:14:13.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-30T11:14:15.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-30T11:14:15.870 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:14:15.871 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-30T11:14:08.136868+0000 21'1 26:66 [1,0] 1 [1,0] 1 21'1 2024-03-30T11:14:07.903042+0000 21'1 2024-03-30T11:14:07.903042+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:14:07.903042+0000 1 0 2024-03-30T11:14:15.871 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:14:15.871 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:14:15.871 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:14:15.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-03-30T11:14:20.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-30T11:14:20.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:14:20.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:14:20.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:14:20.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:14:20.884 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:14:20.884 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:14:20.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:14:20.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:14:20.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:14:20.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:14:20.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:14:20.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:14:20.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:14:20.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:14:20.979 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:14:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:14:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:14:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:14:21.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:14:21.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:21.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:14:21.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-03-30T11:14:21.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-03-30T11:14:21.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-03-30T11:14:21.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:21.380 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:14:21.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-30T11:14:21.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-30T11:14:21.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672980' 2024-03-30T11:14:21.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:21.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:14:21.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-30T11:14:21.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-30T11:14:21.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672980 2-64424509452' 2024-03-30T11:14:21.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:21.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-03-30T11:14:21.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:21.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:14:21.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-03-30T11:14:21.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:21.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-03-30T11:14:21.593 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 107374182407 2024-03-30T11:14:21.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-03-30T11:14:21.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:21.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-03-30T11:14:21.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:14:22.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:14:22.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:23.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-03-30T11:14:23.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:23.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-30T11:14:23.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:23.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:14:23.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-30T11:14:23.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:23.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-30T11:14:23.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-30T11:14:23.175 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672980 2024-03-30T11:14:23.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:14:23.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-30T11:14:23.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:23.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-30T11:14:23.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:23.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:14:23.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-30T11:14:23.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:23.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-30T11:14:23.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-30T11:14:23.478 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509452 2024-03-30T11:14:23.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:14:23.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-03-30T11:14:23.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:14:23.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:14:23.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:14:24.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:14:24.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:14:24.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:14:24.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:14:24.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:14:24.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:14:24.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:14:24.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:14:24.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:14:24.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-30T11:14:25.004 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:14:25.005 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-30T11:14:08.136868+0000 21'1 26:66 [1,0] 1 [1,0] 1 21'1 2024-03-30T11:14:07.903042+0000 21'1 2024-03-30T11:14:07.903042+0000 0 1 periodic scrub scheduled @ 2024-03-31T11:14:07.903042+0000 1 0 2024-03-30T11:14:25.005 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:14:25.005 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:14:25.005 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:14:25.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-30T11:14:25.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:14:25.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:14:25.018 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:14:25.018 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:14:25.018 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:14:25.018 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:14:25.018 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:14:25.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:14:25.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:14:25.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:14:25.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:14:25.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:14:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:14:25.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:14:25.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-30T11:14:27.106 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:14:27.106 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:14:27.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:27.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:14:27.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-30T11:14:27.380 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:14:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:14:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:14:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:14:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:14:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:27.413+0000 7fdfa6212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:27.416+0000 7fdfa6212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:27.412 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:27.417+0000 7fdfa6212640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:27.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:27.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:27.929 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:27.934+0000 7fdfa6212640 -1 Falling back to public interface 2024-03-30T11:14:27.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:28.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:28.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:28.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:14:28.966 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:14:28.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:28.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:29.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:30.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:30.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:30.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:14:30.265 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:14:30.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:30.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:30.289 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:30.294+0000 7fdfa6212640 -1 osd.0 26 log_to_monitors true 2024-03-30T11:14:30.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:31.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:31.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:31.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:14:31.595 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:14:31.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:31.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:31.886 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1124809880,v1:127.0.0.1:6803/1124809880] [v2:127.0.0.1:6804/1124809880,v1:127.0.0.1:6805/1124809880] exists,up a1c8fb56-a78d-4e49-98b7-eba6cd492902 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:14:31.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:14:31.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:14:31.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:14:31.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:14:31.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:14:31.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:14:31.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:14:32.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:14:32.249 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:14:32.250 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:14:32.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:14:32.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:32.250 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:14:32.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-30T11:14:32.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-30T11:14:32.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-30T11:14:32.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:32.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:14:32.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-30T11:14:32.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-30T11:14:32.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983' 2024-03-30T11:14:32.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:14:32.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:14:32.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-30T11:14:32.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-30T11:14:32.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983 2-64424509456' 2024-03-30T11:14:32.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:32.565 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-30T11:14:32.565 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:32.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:14:32.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-30T11:14:32.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:32.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-30T11:14:32.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-30T11:14:32.567 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 128849018882 2024-03-30T11:14:32.567 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:32.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-30T11:14:32.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:14:33.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:14:33.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:14:34.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-30T11:14:34.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:34.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-30T11:14:34.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:34.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:14:34.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-30T11:14:34.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:34.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-30T11:14:34.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-30T11:14:34.182 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672983 2024-03-30T11:14:34.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:14:34.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-30T11:14:34.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:14:34.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-30T11:14:34.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:14:34.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:14:34.468 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-30T11:14:34.468 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:14:34.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-30T11:14:34.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-30T11:14:34.469 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509456 2024-03-30T11:14:34.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:14:34.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-30T11:14:34.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:14:34.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:14:34.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:14:35.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:14:35.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:14:35.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:14:35.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:14:35.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:14:35.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:14:35.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:14:35.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:14:35.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:14:35.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:14:35.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:14:35.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:14:35.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:14:35.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:14:35.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:14:35.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:14:35.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:14:35.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-30T11:14:35.775 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:06.893+0000 7f21796ec640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-03-30T11:14:35.775 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:06.893+0000 7f21796ec640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-03-30T11:14:35.775 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:06.893+0000 7f21796ec640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-30T11:14:35.775 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:07.902+0000 7f21796ec640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:14:35.775 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:07.902+0000 7f21796ec640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:14:35.775 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:07.902+0000 7f21796ec640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-30T11:14:35.776 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:07.902+0000 7f21796ec640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-30T11:14:35.776 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:14:07.902+0000 7f21796ec640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:14:35.776 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-03-30T11:14:35.776 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-03-30T11:14:35.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-03-30T11:14:35.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-03-30T11:14:35.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:14:35.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:14:35.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:14:35.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:14:35.910 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:14:35.910 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:14:35.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:14:35.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:14:35.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:14:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:14:36.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:14:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:14:36.018 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:14:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:14:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:14:36.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:14:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:14:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:14:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:14:36.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:14:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:14:36.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:14:36.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:14:36.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:14:36.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:36.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:36.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:14:36.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:14:36.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:14:36.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:14:36.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:14:36.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:14:36.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:14:36.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:14:36.040 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:14:36.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:14:36.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:14:36.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:14:36.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:14:36.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:14:36.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:14:36.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:14:36.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:14:36.043 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:14:36.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:14:36.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:14:36.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:14:36.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:14:36.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:36.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:36.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:14:36.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:14:36.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:14:36.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:14:36.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:14:36.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:36.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:36.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:14:36.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:14:36.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:14:36.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:14:36.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:36.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:14:36.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:14:36.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:14:36.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:14:36.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:14:36.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:14:36.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:14:36.612 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:14:36.612 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:14:36.612 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:14:36.613 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:14:36.613 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:36.613 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:36.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:14:36.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:14:36.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:14:36.695 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:14:36.695 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:14:36.695 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:14:36.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:14:36.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-03-30T11:14:36.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:14:36.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:14:36.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:14:36.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:14:36.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:14:36.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:14:37.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:14:37.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:14:37.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:14:37.587 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:14:37.587 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:37.587 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:37.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:37.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:14:37.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:14:37.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:14:37.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-03-30T11:14:37.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:14:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:14:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8126c003-485c-4385-97bb-04077528ab95' 2024-03-30T11:14:37.623 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:14:37.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:14:37.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAd9AdmtBxPJhAAecVRSNWlSQEVx6eTN2MD5A== 2024-03-30T11:14:37.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAd9AdmtBxPJhAAecVRSNWlSQEVx6eTN2MD5A=="}' 2024-03-30T11:14:37.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8126c003-485c-4385-97bb-04077528ab95 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:14:37.843 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:14:37.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:14:37.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAd9AdmtBxPJhAAecVRSNWlSQEVx6eTN2MD5A== --osd-uuid 8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:14:37.885 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.890+0000 7f697188b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:37.893 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.899+0000 7f697188b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:37.895 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.900+0000 7f697188b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:37.895 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.900+0000 7f697188b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:14:37.895 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.900+0000 7f697188b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:14:37.895 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.900+0000 7f697188b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:14:37.977 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:37.982+0000 7f697188b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:14:43.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:14:43.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:14:43.560 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:14:43.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:14:43.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:14:43.965 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:14:43.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:14:43.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:14:43.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:14:43.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:14:43.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:14:43.990 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:43.996+0000 7f8763c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:43.992 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:43.998+0000 7f8763c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:43.993 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:43.999+0000 7f8763c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:14:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:14:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:14:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:14:44.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:14:44.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:44.251 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:14:44.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:14:44.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:44.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:44.510 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:44.516+0000 7f8763c22640 -1 Falling back to public interface 2024-03-30T11:14:44.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:45.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:45.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:45.535 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:14:45.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:14:45.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:45.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:45.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:46.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:46.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:46.830 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:14:46.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:14:46.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:46.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:46.842 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:46.847+0000 7f8763c22640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:14:47.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:48.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:48.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:48.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:14:48.186 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:14:48.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:48.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:14:48.478 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/637044485,v1:127.0.0.1:6803/637044485] [v2:127.0.0.1:6804/637044485,v1:127.0.0.1:6805/637044485] exists,up 8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:14:48.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:14:48.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:14:48.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:14:48.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-30T11:14:48.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:14:48.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:14:48.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b8a6bef-2fbb-4eeb-a10f-3f1b715a6761 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 2b8a6bef-2fbb-4eeb-a10f-3f1b715a6761 2024-03-30T11:14:48.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2b8a6bef-2fbb-4eeb-a10f-3f1b715a6761' 2024-03-30T11:14:48.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:14:48.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAo9AdmVOEGHhAAO/KfstHpwRoF2CqGykaveQ== 2024-03-30T11:14:48.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAo9AdmVOEGHhAAO/KfstHpwRoF2CqGykaveQ=="}' 2024-03-30T11:14:48.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b8a6bef-2fbb-4eeb-a10f-3f1b715a6761 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:14:48.870 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:14:48.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:14:48.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAo9AdmVOEGHhAAO/KfstHpwRoF2CqGykaveQ== --osd-uuid 2b8a6bef-2fbb-4eeb-a10f-3f1b715a6761 2024-03-30T11:14:48.913 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:48.918+0000 7f6ccae13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:48.955 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:48.960+0000 7f6ccae13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:48.961+0000 7f6ccae13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:48.962+0000 7f6ccae13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:14:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:48.962+0000 7f6ccae13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:14:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:48.962+0000 7f6ccae13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:14:49.030 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:49.035+0000 7f6ccae13640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:14:54.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:14:54.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:14:54.780 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:14:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:14:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:14:55.148 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:14:55.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:14:55.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:14:55.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:14:55.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:14:55.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:14:55.177 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:55.183+0000 7fb5c0e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:55.179 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:55.185+0000 7fb5c0e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:55.180 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:55.186+0000 7fb5c0e22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:14:55.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:14:55.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:14:55.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:14:55.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:14:55.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:14:55.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:55.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:14:55.432 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:14:55.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:55.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:14:55.692 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:55.698+0000 7fb5c0e22640 -1 Falling back to public interface 2024-03-30T11:14:55.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:14:56.713 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:14:56.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:56.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:14:57.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:58.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:58.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:58.003 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:14:58.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:14:58.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:58.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:14:58.072 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:14:58.078+0000 7fb5c0e22640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:14:58.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:14:59.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:14:59.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:14:59.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:14:59.297 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:14:59.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:14:59.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:14:59.587 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/37366171,v1:127.0.0.1:6811/37366171] [v2:127.0.0.1:6812/37366171,v1:127.0.0.1:6813/37366171] exists,up 2b8a6bef-2fbb-4eeb-a10f-3f1b715a6761 2024-03-30T11:14:59.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:14:59.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:14:59.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:14:59.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:14:59.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42fd22bb-4e6e-444b-83ea-9f1e64c4ce93 2024-03-30T11:14:59.594 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 42fd22bb-4e6e-444b-83ea-9f1e64c4ce93 2024-03-30T11:14:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 42fd22bb-4e6e-444b-83ea-9f1e64c4ce93' 2024-03-30T11:14:59.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:14:59.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAz9Adms9SPJBAAnilXrZbpbp3tPvkI2yJKtw== 2024-03-30T11:14:59.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAz9Adms9SPJBAAnilXrZbpbp3tPvkI2yJKtw=="}' 2024-03-30T11:14:59.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42fd22bb-4e6e-444b-83ea-9f1e64c4ce93 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:14:59.991 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:15:00.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:15:00.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQAz9Adms9SPJBAAnilXrZbpbp3tPvkI2yJKtw== --osd-uuid 42fd22bb-4e6e-444b-83ea-9f1e64c4ce93 2024-03-30T11:15:00.034 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.039+0000 7fc1c2a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.081+0000 7fc1c2a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.082+0000 7fc1c2a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.082+0000 7fc1c2a13640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:15:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.082+0000 7fc1c2a13640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:15:00.077 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.082+0000 7fc1c2a13640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:15:00.151 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:00.156+0000 7fc1c2a13640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:15:06.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:15:06.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:15:06.076 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:15:06.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:15:06.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:15:06.446 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:15:06.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:15:06.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:06.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:15:06.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:15:06.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:15:06.475 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:06.480+0000 7ff137648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:06.477 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:06.482+0000 7ff137648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:06.478 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:06.483+0000 7ff137648640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:06.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:15:06.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:15:06.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:15:06.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:15:06.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:15:06.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:06.738 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:15:06.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:15:06.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:06.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:15:06.993 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:06.999+0000 7ff137648640 -1 Falling back to public interface 2024-03-30T11:15:07.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:08.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:08.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:08.023 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:15:08.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:15:08.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:08.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:15:08.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:09.309 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:09.314+0000 7ff137648640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:15:09.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:09.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:09.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:15:09.316 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:15:09.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:09.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:15:09.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:10.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:10.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:10.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:15:10.617 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:15:10.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:10.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:15:10.908 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3344568752,v1:127.0.0.1:6819/3344568752] [v2:127.0.0.1:6820/3344568752,v1:127.0.0.1:6821/3344568752] exists,up 42fd22bb-4e6e-444b-83ea-9f1e64c4ce93 2024-03-30T11:15:10.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:15:10.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:15:10.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:15:10.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-03-30T11:15:10.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-30T11:15:11.311 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T11:15:11.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:15:12.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-03-30T11:15:12.751 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 2 2024-03-30T11:15:12.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:15:12.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:15:12.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:15:12.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:15:12.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:15:12.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:15:12.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:15:12.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:15:13.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:15:13.277 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:13.277 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:15:13.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:15:13.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:13.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:15:13.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-03-30T11:15:13.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-03-30T11:15:13.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-03-30T11:15:13.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:13.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:15:13.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-03-30T11:15:13.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-03-30T11:15:13.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-03-30T11:15:13.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:13.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:15:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-03-30T11:15:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-03-30T11:15:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-03-30T11:15:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:13.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-03-30T11:15:13.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:13.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:15:13.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-03-30T11:15:13.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:13.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-03-30T11:15:13.730 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836487 2024-03-30T11:15:13.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-03-30T11:15:13.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:14.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836487 2024-03-30T11:15:14.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:15:15.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:15:15.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:15.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836487 2024-03-30T11:15:15.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:15.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-03-30T11:15:15.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:15.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:15:15.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-03-30T11:15:15.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:15.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-03-30T11:15:15.307 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672964 2024-03-30T11:15:15.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-03-30T11:15:15.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:15:15.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-03-30T11:15:15.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:15.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-03-30T11:15:15.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:15.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:15:15.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-03-30T11:15:15.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:15.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-03-30T11:15:15.590 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509442 2024-03-30T11:15:15.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-03-30T11:15:15.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:15.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-03-30T11:15:15.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:15:15.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:15.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:16.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:15:16.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:15:16.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:15:16.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:15:16.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:15:16.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:15:16.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:15:16.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:15:16.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:15:16.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:15:16.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:16.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:16.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:15:16.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:15:16.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:15:16.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-03-30T11:15:16.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-03-30T11:15:16.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:15:17.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-30T11:15:17.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:15:17.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:15:17.000 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:15:17.000 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:15:17.000 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:15:17.001 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:17.001 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:15:17.283 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:15:17.283 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:17.283 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:15:17.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:15:17.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:17.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-30T11:15:19.241 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-30T11:15:19.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:15:19.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:19.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:15:19.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:15:19.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:15:19.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:15:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:19.797+0000 7f169b4b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:19.794 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:19.800+0000 7f169b4b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:19.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:19.801+0000 7f169b4b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:20.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:20.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:20.309 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:20.315+0000 7f169b4b5640 -1 Falling back to public interface 2024-03-30T11:15:20.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:21.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:21.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:21.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:15:21.356 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:21.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:21.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:21.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:22.622 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:22.627+0000 7f169b4b5640 -1 osd.0 21 log_to_monitors true 2024-03-30T11:15:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:15:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:15:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:23.119 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1573055618,v1:127.0.0.1:6803/1573055618] [v2:127.0.0.1:6804/1573055618,v1:127.0.0.1:6805/1573055618] exists,up 8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:15:23.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:15:23.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:15:23.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:15:23.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:15:23.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:15:23.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:15:23.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:15:23.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:15:23.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:15:23.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:15:23.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:15:23.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:15:23.486 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:23.487 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:15:23.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:15:23.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:23.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:15:23.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-30T11:15:23.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-30T11:15:23.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-30T11:15:23.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:23.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:15:23.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-30T11:15:23.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-30T11:15:23.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672967' 2024-03-30T11:15:23.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:23.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:15:23.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-30T11:15:23.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-30T11:15:23.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672967 2-64424509445' 2024-03-30T11:15:23.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:23.837 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-30T11:15:23.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:23.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:15:23.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-30T11:15:23.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:23.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-30T11:15:23.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-30T11:15:23.840 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 107374182402 2024-03-30T11:15:23.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:24.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-30T11:15:24.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:24.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-30T11:15:24.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:24.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:15:24.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-30T11:15:24.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:24.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-30T11:15:24.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-30T11:15:24.186 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672967 2024-03-30T11:15:24.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:15:24.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-30T11:15:24.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:24.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-30T11:15:24.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:24.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:15:24.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-30T11:15:24.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:24.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-30T11:15:24.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-30T11:15:24.471 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509445 2024-03-30T11:15:24.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:24.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509445 2024-03-30T11:15:24.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:15:25.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:15:25.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:26.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-30T11:15:26.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:15:26.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:26.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:15:26.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:15:26.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:15:26.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:15:26.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:15:26.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:26.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:15:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:15:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:15:27.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-03-30T11:15:27.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T11:15:27.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:15:27.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:27.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:15:27.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-03-30T11:15:27.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-03-30T11:15:27.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:15:27.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:15:27.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:27.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:15:27.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-03-30T11:15:27.632 INFO:tasks.workunit.client.0.smithi003.stdout:Affected PG 1.0 w/ primary 1 2024-03-30T11:15:27.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-03-30T11:15:27.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-03-30T11:15:27.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:15:27.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:15:27.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:15:27.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-03-30T11:15:11.254851+0000 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T11:15:27.914 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:15:27.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:15:27.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-03-30T11:15:11.254851+0000 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T11:15:28.191 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T11:15:28.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T11:15:28.281 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T11:15:28.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-03-30T11:15:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T11:15:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-03-30T11:15:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:15:28.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:15:28.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:15:28.653 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:28.653 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:15:28.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:15:28.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:28.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:15:28.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-03-30T11:15:28.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-03-30T11:15:28.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-03-30T11:15:28.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:28.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:15:28.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-30T11:15:28.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-30T11:15:28.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672969' 2024-03-30T11:15:28.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:28.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:15:28.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-30T11:15:28.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-30T11:15:28.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672969 2-64424509447' 2024-03-30T11:15:28.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:28.967 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-03-30T11:15:28.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:28.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:15:28.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-03-30T11:15:28.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:28.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-03-30T11:15:28.970 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 107374182404 2024-03-30T11:15:28.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-03-30T11:15:28.971 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:29.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182404 2024-03-30T11:15:29.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:15:30.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:15:30.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:30.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-03-30T11:15:30.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:30.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-30T11:15:30.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:30.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:15:30.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-30T11:15:30.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:30.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-30T11:15:30.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-30T11:15:30.550 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672969 2024-03-30T11:15:30.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:15:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672969 2024-03-30T11:15:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:30.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-30T11:15:30.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:15:30.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-30T11:15:30.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:30.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-30T11:15:30.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-30T11:15:30.903 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509447 2024-03-30T11:15:30.904 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:31.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-30T11:15:31.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-03-30T11:15:31.185 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30 11:15:31.192306959+00:00 2024-03-30T11:15:31.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-03-30T11:15:31.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T11:15:31.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T11:15:31.186 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:15:31.187 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:31.187 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:31.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T11:15:31.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-03-30T11:15:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-03-30T11:15:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-30T11:15:31.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-30T11:15:31.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-30T11:15:31.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-30T11:15:31.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-03-30T11:15:31.387 INFO:tasks.workunit.client.0.smithi003.stdout:{"deep":true,"must":false,"stamp":"2024-03-16T11:13:51.277425+0000"}0 ) state now: "active+clean" 2024-03-30T11:15:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-30T11:15:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-30T11:15:31.388 INFO:tasks.workunit.client.0.smithi003.stdout:"active+clean" 2024-03-30T11:15:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-03-30T11:15:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-30T11:15:31.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-30T11:15:31.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-30T11:15:31.689 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-30T11:15:31.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-30T11:15:31.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-30T11:15:31.826 INFO:tasks.workunit.client.0.smithi003.stdout:1 ) state now: "active+clean" 2024-03-30T11:15:31.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-03-30T11:15:31.827 INFO:tasks.workunit.client.0.smithi003.stdout:"active+clean" 2024-03-30T11:15:31.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-30T11:15:31.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-30T11:15:31.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-03-30T11:15:31.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-30T11:15:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-30T11:15:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-30T11:15:32.127 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-30T11:15:32.127 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-30T11:15:32.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-03-30T11:15:32.254 INFO:tasks.workunit.client.0.smithi003.stdout:2 ) state now: "active+clean+scrubbing+deep" 2024-03-30T11:15:32.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean+scrubbing+deep"' 2024-03-30T11:15:32.255 INFO:tasks.workunit.client.0.smithi003.stdout:found scrub 2024-03-30T11:15:32.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-30T11:15:32.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-03-30T11:15:32.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-03-30T11:15:32.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-03-30T11:15:32.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-03-30T11:15:32.256 INFO:tasks.workunit.client.0.smithi003.stdout:initiate_and_fetch_state ret: 0 2024-03-30T11:15:32.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-03-30T11:15:32.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T11:15:32.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.td/osd-scrub-repair.asok 2024-03-30T11:15:32.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-03-30T11:15:32.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-03-30T11:15:32.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-03-30T11:15:32.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-30T11:15:32.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-30T11:15:32.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-30T11:15:32.432 INFO:tasks.workunit.client.0.smithi003.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-30T11:15:32.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-30T11:15:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-03-30T11:15:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-30T11:15:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-30T11:15:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-30T11:15:32.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-30T11:15:32.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-30T11:15:32.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-30T11:15:32.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-30T11:15:32.844 INFO:tasks.workunit.client.0.smithi003.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-30T11:15:32.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:32.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-03-30T11:15:32.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-30T11:15:33.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-30T11:15:33.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-30T11:15:33.145 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-30T11:15:33.145 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-30T11:15:33.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-30T11:15:33.256 INFO:tasks.workunit.client.0.smithi003.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-30T11:15:33.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-30T11:15:33.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:33.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-03-30T11:15:33.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-30T11:15:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-30T11:15:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-30T11:15:33.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-30T11:15:33.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-30T11:15:33.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-30T11:15:33.667 INFO:tasks.workunit.client.0.smithi003.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-30T11:15:33.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-30T11:15:33.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:33.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-03-30T11:15:33.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-30T11:15:33.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-30T11:15:33.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-30T11:15:33.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-30T11:15:33.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-30T11:15:34.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-30T11:15:34.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-30T11:15:34.078 INFO:tasks.workunit.client.0.smithi003.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-30T11:15:34.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:34.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-03-30T11:15:34.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-03-30T11:15:34.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:15:34.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:15:34.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:15:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:15:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:15:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:34.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:15:34.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-03-30T11:15:34.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-03-30T11:15:34.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-03-30T11:15:34.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:34.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:15:34.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-30T11:15:34.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-30T11:15:34.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672972' 2024-03-30T11:15:34.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:34.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:15:34.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-30T11:15:34.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-30T11:15:34.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672972 2-64424509449' 2024-03-30T11:15:34.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:34.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-03-30T11:15:34.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:34.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:15:34.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-03-30T11:15:34.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:34.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-03-30T11:15:34.678 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 107374182406 2024-03-30T11:15:34.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-03-30T11:15:34.679 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:34.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182406 2024-03-30T11:15:34.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:15:35.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:15:35.972 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:36.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-03-30T11:15:36.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:36.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-30T11:15:36.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:36.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:15:36.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-30T11:15:36.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:36.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-30T11:15:36.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-30T11:15:36.267 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672972 2024-03-30T11:15:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:15:36.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-30T11:15:36.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:36.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-30T11:15:36.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:36.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:15:36.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-30T11:15:36.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:36.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-30T11:15:36.549 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509449 2024-03-30T11:15:36.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-30T11:15:36.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:36.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-30T11:15:36.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-30T11:15:37.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-30T11:15:37.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-30T11:15:37.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-30T11:15:37.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-30T11:15:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-03-30T11:15:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-03-30T11:15:37.255 INFO:tasks.workunit.client.0.smithi003.stdout:wait-scrub-end state now: "active+clean" 2024-03-30T11:15:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-03-30T11:15:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-30T11:15:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-03-30T11:15:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-03-30T11:15:37.520 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T11:15:37.521 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-03-30T11:15:31.972795+0000 21'1 26:44 [1,0] 1 [1,0] 1 0'0 2024-03-16T11:13:51.277425+0000 0'0 2024-03-16T11:13:51.277425+0000 0 0 deep scrubbing for 2s 0 0 2024-03-30T11:15:37.521 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T11:15:37.521 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T11:15:37.521 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:37.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:15:37.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:15:37.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:37.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:15:38.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:15:38.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:15:38.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:15:38.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:15:38.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:15:38.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:15:38.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-30T11:15:38.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:15:38.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:15:38.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:15:38.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:15:38.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:38.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-30T11:15:39.586 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:15:39.586 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:15:39.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:39.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:15:39.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:15:39.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:15:39.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:15:39.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:15:39.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:15:39.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:15:39.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:15:39.856 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:15:39.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:39.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:15:39.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:15:39.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:15:39.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:15:39.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:15:39.886 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:39.891+0000 7f32856d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:39.888 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:39.894+0000 7f32856d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:39.889 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:39.895+0000 7f32856d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:40.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:40.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:40.407 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:40.413+0000 7f32856d9640 -1 Falling back to public interface 2024-03-30T11:15:40.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:15:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:41.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:42.628 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:42.633+0000 7f32856d9640 -1 osd.0 26 log_to_monitors true 2024-03-30T11:15:42.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:42.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:42.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:15:42.720 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:15:42.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:42.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1948155970,v1:127.0.0.1:6803/1948155970] [v2:127.0.0.1:6804/1948155970,v1:127.0.0.1:6805/1948155970] exists,up 8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:15:43.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:15:43.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:15:43.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:15:43.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:15:43.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:15:43.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:15:43.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:15:43.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:15:43.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:15:43.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:15:43.432 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:43.433 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:15:43.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:15:43.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:43.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:15:43.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-30T11:15:43.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-30T11:15:43.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-30T11:15:43.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:43.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:15:43.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-30T11:15:43.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-30T11:15:43.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672974' 2024-03-30T11:15:43.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:43.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:15:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-30T11:15:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-30T11:15:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672974 2-64424509452' 2024-03-30T11:15:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:43.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-30T11:15:43.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:15:43.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-30T11:15:43.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:43.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-30T11:15:43.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-30T11:15:43.753 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 128849018882 2024-03-30T11:15:43.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:44.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-30T11:15:44.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:44.034 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-30T11:15:44.034 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:44.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:15:44.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-30T11:15:44.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:44.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-30T11:15:44.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-30T11:15:44.037 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672974 2024-03-30T11:15:44.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:15:44.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-30T11:15:44.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:44.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-30T11:15:44.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:44.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:15:44.430 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-30T11:15:44.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:44.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-30T11:15:44.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-30T11:15:44.432 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509452 2024-03-30T11:15:44.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:44.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509452 2024-03-30T11:15:44.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:15:45.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:15:45.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:46.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-30T11:15:46.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:15:46.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:46.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:15:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:15:46.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:15:46.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:15:46.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:15:46.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:15:46.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:15:46.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:15:46.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:15:46.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:15:46.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:46.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:47.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:15:47.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:15:47.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-30T11:15:47.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:15:47.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:15:47.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:15:47.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:15:47.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:15:47.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:15:47.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:15:47.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:15:47.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:15:47.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:15:47.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:15:47.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:15:47.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:15:47.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:47.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-30T11:15:49.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:15:49.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:15:49.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:15:49.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-30T11:15:49.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:15:49.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:15:49.464 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:49.470+0000 7fccb8011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:49.466 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:49.472+0000 7fccb8011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:49.468 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:49.473+0000 7fccb8011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:49.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:15:49.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:15:49.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:15:49.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:15:49.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:15:49.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:49.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:15:49.726 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:15:49.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:49.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:49.985 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:49.991+0000 7fccb8011640 -1 Falling back to public interface 2024-03-30T11:15:50.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:51.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:51.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:51.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:15:51.032 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:51.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:51.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:52.288 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:52.293+0000 7fccb8011640 -1 osd.0 31 log_to_monitors true 2024-03-30T11:15:52.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:52.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:52.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:15:52.324 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:15:52.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:52.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:15:53.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:15:53.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:15:53.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:15:53.620 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:15:53.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:15:53.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/4001644045,v1:127.0.0.1:6803/4001644045] [v2:127.0.0.1:6804/4001644045,v1:127.0.0.1:6805/4001644045] exists,up 8126c003-485c-4385-97bb-04077528ab95 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:15:53.911 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:15:53.912 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:15:53.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:15:53.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:15:53.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:15:53.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:15:53.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:15:53.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:15:53.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:15:53.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:15:53.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:15:54.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:15:54.276 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:15:54.276 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:15:54.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:15:54.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:54.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:15:54.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-30T11:15:54.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-30T11:15:54.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-30T11:15:54.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:54.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:15:54.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-30T11:15:54.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-30T11:15:54.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672977' 2024-03-30T11:15:54.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:15:54.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:15:54.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-30T11:15:54.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-30T11:15:54.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672977 2-64424509455' 2024-03-30T11:15:54.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:54.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-30T11:15:54.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:54.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:15:54.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-30T11:15:54.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:54.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-30T11:15:54.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-30T11:15:54.594 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 150323855362 2024-03-30T11:15:54.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:54.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-30T11:15:54.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:15:55.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:15:55.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:15:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-30T11:15:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:56.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-30T11:15:56.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:56.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:15:56.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-30T11:15:56.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-30T11:15:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-30T11:15:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672977 2024-03-30T11:15:56.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:15:56.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-03-30T11:15:56.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:15:56.458 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-30T11:15:56.458 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:15:56.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:15:56.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-30T11:15:56.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:15:56.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-30T11:15:56.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-30T11:15:56.461 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509455 2024-03-30T11:15:56.461 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:15:56.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-30T11:15:56.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:15:56.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:56.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:57.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T11:15:57.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:15:57.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:15:57.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:15:57.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:15:57.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:15:57.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:15:57.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:15:57.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:15:57.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:15:57.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:15:57.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:15:57.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T11:15:57.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:15:57.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:15:57.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:15:57.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-30T11:15:57.808 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:15:34.973+0000 7fb5a6ee3640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-30T11:15:57.808 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:15:34.973+0000 7fb5a6ee3640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-30T11:15:57.808 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:15:34.973+0000 7fb5a6ee3640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-30T11:15:57.809 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:15:34.973+0000 7fb5a6ee3640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-30T11:15:57.809 INFO:tasks.workunit.client.0.smithi003.stdout:2024-03-30T11:15:34.973+0000 7fb5a6ee3640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:15:57.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:15:57.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:15:57.917 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:15:57.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:15:57.918 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:15:57.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:15:57.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:15:57.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:15:57.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:15:57.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:15:57.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:15:57.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:15:57.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:15:57.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:15:57.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:15:57.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:15:57.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:57.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:57.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:15:57.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:15:57.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:15:57.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:15:57.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:15:57.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:15:57.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:15:57.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:15:57.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:15:57.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:15:57.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:15:57.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:15:57.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:15:57.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:15:57.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:15:57.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:15:57.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:15:57.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:15:57.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:57.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:57.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:15:57.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:15:57.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:15:57.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:15:57.942 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:15:57.942 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:57.942 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:57.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:15:57.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:15:57.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:15:57.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:15:57.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:15:57.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:15:57.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:15:58.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:15:58.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:15:58.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:15:58.328 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:15:58.328 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:58.328 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:58.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:58.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:15:58.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:15:58.504 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:58.505 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:15:58.506 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:15:58.506 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:15:58.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:15:58.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:15:58.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:15:58.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:15:58.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:15:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:15:59.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:15:59.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:15:59.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:15:59.488 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:15:59.489 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:59.489 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:59.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:59.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:15:59.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:15:59.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:15:59.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-30T11:15:59.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-30T11:15:59.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:15:59.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:15:59.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:15:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:15:59.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:15:59.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:15:59.525 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:15:59.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f4286229-9971-4bcf-9742-6c73d32c7293 2024-03-30T11:15:59.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f4286229-9971-4bcf-9742-6c73d32c7293' 2024-03-30T11:15:59.525 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 f4286229-9971-4bcf-9742-6c73d32c7293 2024-03-30T11:15:59.526 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:15:59.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBv9Admobt4IBAAxgoEeHzquXPDX4X7pKoLBw== 2024-03-30T11:15:59.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBv9Admobt4IBAAxgoEeHzquXPDX4X7pKoLBw=="}' 2024-03-30T11:15:59.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f4286229-9971-4bcf-9742-6c73d32c7293 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:15:59.743 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:15:59.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:15:59.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBv9Admobt4IBAAxgoEeHzquXPDX4X7pKoLBw== --osd-uuid f4286229-9971-4bcf-9742-6c73d32c7293 2024-03-30T11:15:59.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.791+0000 7eff112b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:59.794 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.800+0000 7eff112b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:59.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.801+0000 7eff112b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:15:59.795 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.801+0000 7eff112b5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:15:59.796 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.801+0000 7eff112b5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:15:59.796 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.801+0000 7eff112b5640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:15:59.886 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:15:59.891+0000 7eff112b5640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:16:05.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:16:05.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:16:05.638 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:16:05.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:16:05.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:16:06.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:16:06.009 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:16:06.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:16:06.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:16:06.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:16:06.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:16:06.038 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:06.044+0000 7f7a22c23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:06.040 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:06.046+0000 7f7a22c23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:06.041 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:06.047+0000 7f7a22c23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:16:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:16:06.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:06.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:06.562 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:06.568+0000 7f7a22c23640 -1 Falling back to public interface 2024-03-30T11:16:06.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:07.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:07.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:07.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:16:07.570 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:16:07.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:07.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:07.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:08.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:08.865 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:16:08.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:08.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:16:08.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:08.901 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:08.906+0000 7f7a22c23640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:16:09.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:10.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:10.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:10.168 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:16:10.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:16:10.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:10.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:10.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:11.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:11.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:11.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:16:11.472 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:16:11.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:11.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:12.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:12.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:16:12.849 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:16:12.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:12.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:13.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:14.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:14.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:14.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:16:14.146 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:16:14.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:14.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:14.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:15.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:15.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:15.488 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:16:15.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:16:15.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:15.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:15.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:16.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:16.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:16.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:16:16.877 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:16:16.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:16.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:17.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:18.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:18.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:18.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:16:18.173 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:16:18.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:18.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:18.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:19.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:19.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:19.480 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:16:19.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:16:19.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:19.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:19.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:20.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:20.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:20.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:16:20.873 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:16:20.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:20.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:21.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:22.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:22.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:22.220 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:16:22.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:16:22.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:22.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:22.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:23.817 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:16:23.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:16:23.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:23.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:24.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:16:25.435 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:16:25.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:25.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:25.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:26.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:26.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:26.829 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:16:26.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:16:26.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:26.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:27.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:28.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:28.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:28.417 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:16:28.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:16:28.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:28.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:29.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:30.110 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:16:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:16:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:30.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:31.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:31.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:31.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:16:31.737 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:16:31.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:31.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:16:32.030 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1394649785,v1:127.0.0.1:6803/1394649785] [v2:127.0.0.1:6804/1394649785,v1:127.0.0.1:6805/1394649785] exists,up f4286229-9971-4bcf-9742-6c73d32c7293 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:16:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:16:32.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:16:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8754c6c1-f6ff-461e-b910-5a5ed0cddbbd 2024-03-30T11:16:32.037 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd 2024-03-30T11:16:32.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd' 2024-03-30T11:16:32.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:16:32.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQ9AdmOkFiAxAA4oQeCld3jCfhB+3fZxa/3w== 2024-03-30T11:16:32.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQ9AdmOkFiAxAA4oQeCld3jCfhB+3fZxa/3w=="}' 2024-03-30T11:16:32.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd -i td/osd-scrub-repair/1/new.json 2024-03-30T11:16:32.421 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:16:32.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:16:32.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCQ9AdmOkFiAxAA4oQeCld3jCfhB+3fZxa/3w== --osd-uuid 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd 2024-03-30T11:16:32.464 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.469+0000 7fedeb36f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:32.505 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.511+0000 7fedeb36f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:32.506 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.512+0000 7fedeb36f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:32.507 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.512+0000 7fedeb36f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:16:32.507 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.512+0000 7fedeb36f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:16:32.507 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.512+0000 7fedeb36f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:16:32.572 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:32.577+0000 7fedeb36f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:16:38.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:16:38.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:16:38.355 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:16:38.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:16:38.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:16:38.723 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:16:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:16:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:16:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:16:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:16:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:16:38.753 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:38.758+0000 7f20faa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:38.754 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:38.760+0000 7f20faa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:38.756 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:38.761+0000 7f20faa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:16:39.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:16:39.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:16:39.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:16:39.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:16:39.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:16:39.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:39.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:16:39.027 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:16:39.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:39.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:39.287 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:39.293+0000 7f20faa11640 -1 Falling back to public interface 2024-03-30T11:16:39.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:40.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:40.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:40.313 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:16:40.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:16:40.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:40.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:40.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:41.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:16:41.610 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:16:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:41.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:41.706 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:16:41.712+0000 7f20faa11640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:16:41.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:42.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:42.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:42.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:16:42.978 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:16:42.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:42.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:43.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:44.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:44.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:44.271 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:16:44.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:16:44.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:44.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:44.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:45.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:45.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:45.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:16:45.628 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:16:45.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:45.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:45.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:46.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:46.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:46.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:16:46.969 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:16:46.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:46.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:47.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:48.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:48.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:48.282 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:16:48.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:16:48.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:48.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:48.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:49.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:49.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:49.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:16:49.577 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:16:49.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:49.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:49.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:50.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:50.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:50.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:16:50.871 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:16:50.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:50.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:51.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:52.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:52.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:52.207 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:16:52.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:16:52.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:52.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:52.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:53.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:53.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:53.502 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:16:53.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:16:53.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:53.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:53.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:54.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:54.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:54.984 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:16:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:16:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:55.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:16:56.453 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:16:56.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:56.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:57.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:58.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:58.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:58.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:16:58.038 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:16:58.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:58.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:16:58.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:16:59.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:16:59.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:16:59.732 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:16:59.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:16:59.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:16:59.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:17:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:01.735 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:17:01.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:01.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:01.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:17:01.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:01.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:17:02.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:03.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:03.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:03.488 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:17:03.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:17:03.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:03.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:17:03.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:17:04.965 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:17:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:17:05.278 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3639155770,v1:127.0.0.1:6811/3639155770] [v2:127.0.0.1:6812/3639155770,v1:127.0.0.1:6813/3639155770] exists,up 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd 2024-03-30T11:17:05.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:17:05.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:17:05.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d40f834f-33d5-4d8b-a1cd-9308fa82a7d1 2024-03-30T11:17:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d40f834f-33d5-4d8b-a1cd-9308fa82a7d1' 2024-03-30T11:17:05.286 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 d40f834f-33d5-4d8b-a1cd-9308fa82a7d1 2024-03-30T11:17:05.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:17:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCx9Adm71wuEhAAyBOSWo2euKFKtsFUgiPhAg== 2024-03-30T11:17:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCx9Adm71wuEhAAyBOSWo2euKFKtsFUgiPhAg=="}' 2024-03-30T11:17:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d40f834f-33d5-4d8b-a1cd-9308fa82a7d1 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:17:05.659 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:17:05.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:17:05.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCx9Adm71wuEhAAyBOSWo2euKFKtsFUgiPhAg== --osd-uuid d40f834f-33d5-4d8b-a1cd-9308fa82a7d1 2024-03-30T11:17:05.702 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.707+0000 7f2ca62ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:05.743 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.749+0000 7f2ca62ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:05.744 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.750+0000 7f2ca62ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:05.745 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.750+0000 7f2ca62ac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:17:05.745 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.750+0000 7f2ca62ac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:17:05.745 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.750+0000 7f2ca62ac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:17:05.818 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:05.824+0000 7f2ca62ac640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:17:11.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:17:11.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:17:11.804 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:17:11.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:17:11.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:17:12.172 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:17:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:17:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:17:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:17:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:17:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:17:12.201 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:12.207+0000 7fc5af4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:12.203 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:12.209+0000 7fc5af4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:12.204 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:12.210+0000 7fc5af4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:17:12.452 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:17:12.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:12.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:12.725 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:12.731+0000 7fc5af4d9640 -1 Falling back to public interface 2024-03-30T11:17:12.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:13.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:13.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:13.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:17:13.750 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:17:13.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:13.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:14.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:15.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:15.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:15.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:17:15.042 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:17:15.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:15.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:15.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:15.522 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:15.528+0000 7fc5af4d9640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:17:16.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:16.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:16.413 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:17:16.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:17:16.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:16.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:16.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:17.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:17.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:17.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:17:17.733 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:17:17.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:17.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:18.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:19.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:19.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:19.029 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:17:19.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:17:19.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:19.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:19.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:20.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:20.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:20.388 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:17:20.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:17:20.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:20.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:20.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:21.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:21.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:21.821 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:17:21.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:17:21.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:21.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:22.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:23.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:23.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:23.115 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:17:23.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:17:23.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:23.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:23.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:24.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:24.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:24.409 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:17:24.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:17:24.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:24.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:24.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:25.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:25.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:25.726 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:17:25.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:17:25.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:25.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:26.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:27.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:27.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:27.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:17:27.139 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:17:27.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:27.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:27.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:28.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:28.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:28.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:17:28.532 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:17:28.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:28.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:29.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:30.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:30.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:30.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:17:30.277 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:17:30.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:30.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:31.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:32.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:32.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:32.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:17:32.033 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:17:32.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:32.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:32.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:33.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:33.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:33.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:17:33.840 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:17:33.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:33.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:34.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:35.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:35.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:35.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:17:35.157 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:17:35.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:35.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:35.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:36.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:36.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:36.602 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:17:36.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:17:36.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:36.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:37.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:38.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:38.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:38.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:17:38.286 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:17:38.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:38.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:38.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:39.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:39.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:39.699 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:17:39.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:17:39.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:39.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:17:40.000 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2752720693,v1:127.0.0.1:6819/2752720693] [v2:127.0.0.1:6820/2752720693,v1:127.0.0.1:6821/2752720693] exists,up d40f834f-33d5-4d8b-a1cd-9308fa82a7d1 2024-03-30T11:17:40.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:17:40.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:17:40.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:17:40.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-30T11:17:40.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:17:40.280 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:17:40.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:17:40.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:17:40.715 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:17:40.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:17:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:17:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-30T11:17:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:17:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:17:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:17:44.575 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:17:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:17:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:17:44.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:17:44.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:17:44.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:17:44.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:17:44.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:17:44.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:17:44.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:17:44.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:17:44.671 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:17:44.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:17:44.957 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:17:44.957 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:17:44.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:17:44.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:17:44.957 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:17:45.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-03-30T11:17:45.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-03-30T11:17:45.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-03-30T11:17:45.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:17:45.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:17:45.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-30T11:17:45.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-30T11:17:45.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-64424509457' 2024-03-30T11:17:45.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:17:45.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:17:45.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953479 2024-03-30T11:17:45.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953479 2024-03-30T11:17:45.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-64424509457 2-137438953479' 2024-03-30T11:17:45.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:17:45.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-03-30T11:17:45.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:17:45.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:17:45.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-03-30T11:17:45.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:17:45.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-03-30T11:17:45.274 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836496 2024-03-30T11:17:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-03-30T11:17:45.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:17:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-03-30T11:17:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:17:46.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:17:46.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:17:46.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-03-30T11:17:46.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:17:46.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509457 2024-03-30T11:17:46.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:17:46.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:17:46.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509457 2024-03-30T11:17:46.886 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:17:46.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-30T11:17:46.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509457' 2024-03-30T11:17:46.887 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 64424509457 2024-03-30T11:17:46.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:17:47.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509457 2024-03-30T11:17:47.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:17:47.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953479 2024-03-30T11:17:47.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:17:47.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:17:47.169 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953479 2024-03-30T11:17:47.169 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:17:47.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953479 2024-03-30T11:17:47.170 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 137438953479 2024-03-30T11:17:47.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953479' 2024-03-30T11:17:47.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:17:47.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953479 -lt 137438953479 2024-03-30T11:17:47.451 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:17:47.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:17:47.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:17:47.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:17:47.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:17:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:17:48.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:17:48.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:17:48.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:17:48.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:17:48.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:17:48.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:17:48.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-30T11:17:48.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:17:48.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:17:48.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-30T11:17:48.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:17:48.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-30T11:17:49.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:17:49.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:17:49.936 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:17:49.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:17:50.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-30T11:17:50.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:17:50.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:17:50.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:17:50.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:17:50.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:17:50.951 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:17:50.951 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:17:50.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:17:50.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:17:50.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:17:51.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:17:51.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:17:51.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:17:51.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:17:51.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:17:51.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:17:51.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:17:51.327 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:17:51.327 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:17:51.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:17:51.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:17:51.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:17:51.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-30T11:17:51.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-30T11:17:51.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-30T11:17:51.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:17:51.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:17:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-30T11:17:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-30T11:17:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-64424509460' 2024-03-30T11:17:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:17:51.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:17:51.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953481 2024-03-30T11:17:51.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953481 2024-03-30T11:17:51.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-64424509460 2-137438953481' 2024-03-30T11:17:51.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:17:51.655 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-30T11:17:51.655 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:17:51.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:17:51.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-30T11:17:51.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:17:51.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-30T11:17:51.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-30T11:17:51.657 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836499 2024-03-30T11:17:51.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:17:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-30T11:17:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:17:51.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509460 2024-03-30T11:17:51.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:17:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:17:51.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509460 2024-03-30T11:17:51.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:17:51.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-30T11:17:51.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509460' 2024-03-30T11:17:51.943 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 64424509460 2024-03-30T11:17:51.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:17:52.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-30T11:17:52.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:17:52.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953481 2024-03-30T11:17:52.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:17:52.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:17:52.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953481 2024-03-30T11:17:52.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:17:52.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953481 2024-03-30T11:17:52.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953481' 2024-03-30T11:17:52.229 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 137438953481 2024-03-30T11:17:52.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:17:52.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953480 -lt 137438953481 2024-03-30T11:17:52.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:17:53.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:17:53.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:17:53.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953481 -lt 137438953481 2024-03-30T11:17:53.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:17:53.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:17:53.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:17:54.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:17:54.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:17:54.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:17:54.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:17:54.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:17:54.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:17:54.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:17:54.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:17:54.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:17:54.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-30T11:17:54.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-30T11:17:54.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:17:54.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:17:54.985 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:17:55.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-30T11:17:55.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:17:55.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-30T11:17:55.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:17:55.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:17:55.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:17:57.622 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:17:57.622 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-30T11:17:57.622 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:17:58.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:17:58.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:17:58.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:17:58.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:17:58.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:17:58.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:17:58.141 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:17:58.142 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:17:58.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:17:58.143 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:17:58.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:17:58.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:17:58.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:17:58.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:17:58.173 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:58.178+0000 7f0c13618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:58.175 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:58.181+0000 7f0c13618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:58.176 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:58.182+0000 7f0c13618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:17:58.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:58.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:17:58.752 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:17:58.697+0000 7f0c13618640 -1 Falling back to public interface 2024-03-30T11:17:58.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:17:59.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:17:59.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:17:59.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:17:59.736 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:17:59.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:17:59.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:00.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:01.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:01.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:01.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:18:01.040 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:18:01.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:01.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:01.322 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:01.328+0000 7f0c13618640 -1 osd.1 55 log_to_monitors true 2024-03-30T11:18:01.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:02.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:02.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:02.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:18:02.334 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:18:02.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:02.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 56 last_clean_interval [15,55) [v2:127.0.0.1:6810/1341624906,v1:127.0.0.1:6811/1341624906] [v2:127.0.0.1:6812/1341624906,v1:127.0.0.1:6813/1341624906] exists,up 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:18:02.768 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:18:02.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:18:02.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:18:02.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:18:02.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:18:02.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:18:02.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:18:02.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:18:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:18:02.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:18:03.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:18:03.157 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:18:03.157 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:18:03.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:18:03.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:03.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:18:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-30T11:18:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-30T11:18:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-30T11:18:03.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:03.288 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:18:03.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972354 2024-03-30T11:18:03.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972354 2024-03-30T11:18:03.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-266287972354' 2024-03-30T11:18:03.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:03.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:18:03.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953484 2024-03-30T11:18:03.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953484 2024-03-30T11:18:03.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-266287972354 2-137438953484' 2024-03-30T11:18:03.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:03.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-30T11:18:03.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:03.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:18:03.545 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-30T11:18:03.545 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:03.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-30T11:18:03.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-30T11:18:03.547 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836502 2024-03-30T11:18:03.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:18:03.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-30T11:18:03.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:03.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972354 2024-03-30T11:18:03.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:03.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:18:03.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972354 2024-03-30T11:18:03.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:03.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972354 2024-03-30T11:18:03.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972354' 2024-03-30T11:18:03.867 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 266287972354 2024-03-30T11:18:03.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:18:04.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972354 -lt 266287972354 2024-03-30T11:18:04.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:04.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953484 2024-03-30T11:18:04.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:04.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:18:04.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953484 2024-03-30T11:18:04.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:04.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953484 2024-03-30T11:18:04.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953484' 2024-03-30T11:18:04.157 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 137438953484 2024-03-30T11:18:04.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:18:04.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953484 -lt 137438953484 2024-03-30T11:18:04.440 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:18:04.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:04.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:18:04.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:18:05.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:18:05.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:18:05.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:05.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:05.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:18:05.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:18:05.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:18:05.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:18:05.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:18:05.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:18:05.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:18:05.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:18:05.578 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:18:05.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:18:05.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:18:05.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:18:05.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:18:05.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:18:05.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:18:05.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:18:05.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:05.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:06.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:18:06.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:18:06.289 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:18:06.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:18:06.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:18:06.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:18:06.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:18:06.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:18:06.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:18:06.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:18:06.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:18:06.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:18:06.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:18:06.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:18:06.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:18:06.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:18:07.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:18:07.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:18:07.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:07.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:18:07.502 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:18:07.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:18:07.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:18:07.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T11:18:07.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:18:07.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T11:18:08.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:18:08.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:18:08.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:18:08.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:18:08.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:18:08.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:18:08.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:18:08.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:18:08.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:18:08.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:18:08.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:08.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:08.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:18:08.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:18:08.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:18:08.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-30T11:18:08.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:18:08.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:18:08.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:18:08.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:18:09.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-30T11:18:09.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-30T11:18:09.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:18:09.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:18:09.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:18:09.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:18:09.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-30T11:17:49.849992+0000 2024-03-30T11:18:09.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:18:09.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:17:49.849992+0000 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:18:09.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:18:09.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:18:06.979055+0000 '>' 2024-03-30T11:17:49.849992+0000 2024-03-30T11:18:09.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:18:09.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-30T11:18:09.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:18:09.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:18:09.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:18:09.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:18:09.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:18:09.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:18:09.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:18:09.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:18:09.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:18:10.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:18:10.171 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:18:10.171 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:18:10.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:18:10.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:10.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:18:10.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-30T11:18:10.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-30T11:18:10.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-30T11:18:10.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:10.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:18:10.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972356 2024-03-30T11:18:10.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972356 2024-03-30T11:18:10.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-266287972356' 2024-03-30T11:18:10.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:10.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:18:10.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953487 2024-03-30T11:18:10.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953487 2024-03-30T11:18:10.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-266287972356 2-137438953487' 2024-03-30T11:18:10.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:10.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-30T11:18:10.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:10.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:18:10.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-30T11:18:10.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:10.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-30T11:18:10.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-30T11:18:10.490 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836504 2024-03-30T11:18:10.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:18:10.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-30T11:18:10.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:18:11.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:18:11.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:18:12.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836504 2024-03-30T11:18:12.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:12.091 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972356 2024-03-30T11:18:12.091 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:12.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:18:12.092 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972356 2024-03-30T11:18:12.092 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:12.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972356 2024-03-30T11:18:12.093 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 266287972356 2024-03-30T11:18:12.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972356' 2024-03-30T11:18:12.094 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:18:12.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972357 -lt 266287972356 2024-03-30T11:18:12.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:12.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953487 2024-03-30T11:18:12.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:12.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:18:12.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953487 2024-03-30T11:18:12.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:12.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953487 2024-03-30T11:18:12.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953487' 2024-03-30T11:18:12.404 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 137438953487 2024-03-30T11:18:12.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:18:12.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953487 -lt 137438953487 2024-03-30T11:18:12.687 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:18:12.687 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:12.687 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:13.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:18:13.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:18:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:18:13.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:18:13.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:13.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:13.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:18:13.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:18:13.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:18:13.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-30T11:18:13.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-30T11:18:13.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:18:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-30T11:18:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:18:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:18:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:18:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:18:14.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-30T11:18:14.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:18:14.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-30T11:18:14.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:18:14.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:18:14.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:18:14.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:18:14.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:18:14.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:18:14.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:18:14.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:18:14.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:18:14.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:18:14.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:18:14.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:18:14.334 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:18:14.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:18:14.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:18:14.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:18:15.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:18:15.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:18:17.044 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:18:17.044 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-30T11:18:17.044 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:18:17.044 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:18:17.044 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:18:17.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:18:17.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:18:17.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:17.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:18:17.326 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:18:17.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:18:17.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:18:17.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:18:17.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:18:17.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:18:17.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:18:17.357 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:17.362+0000 7f4df2d21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:17.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:17.365+0000 7f4df2d21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:17.360 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:17.366+0000 7f4df2d21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:18:17.618 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:18:17.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:17.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:17.884 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:17.890+0000 7f4df2d21640 -1 Falling back to public interface 2024-03-30T11:18:17.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:18.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:18.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:18.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:18:18.917 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:18:18.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:18.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:19.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:20.175 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:20.180+0000 7f4df2d21640 -1 osd.1 69 log_to_monitors true 2024-03-30T11:18:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:20.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:20.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:18:20.213 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:18:20.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:20.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:20.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:21.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:21.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:21.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:18:21.565 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:18:21.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:21.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:18:21.858 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 71 last_clean_interval [62,69) [v2:127.0.0.1:6810/2417648207,v1:127.0.0.1:6811/2417648207] [v2:127.0.0.1:6812/2417648207,v1:127.0.0.1:6813/2417648207] exists,up 8754c6c1-f6ff-461e-b910-5a5ed0cddbbd 2024-03-30T11:18:21.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:18:21.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:18:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:18:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:18:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:18:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:18:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:18:21.942 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:18:22.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:18:22.282 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:18:22.282 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:18:22.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:18:22.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:22.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:18:22.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-30T11:18:22.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-30T11:18:22.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-30T11:18:22.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:22.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:18:22.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:18:22.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:18:22.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-330712481794' 2024-03-30T11:18:22.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:18:22.577 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:18:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953490 2024-03-30T11:18:22.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953490 2024-03-30T11:18:22.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-330712481794 2-137438953490' 2024-03-30T11:18:22.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:22.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-30T11:18:22.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:22.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:18:22.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-30T11:18:22.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:22.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-30T11:18:22.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-30T11:18:22.770 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836508 2024-03-30T11:18:22.770 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:18:23.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-30T11:18:23.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:18:24.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:18:24.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:18:24.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-30T11:18:24.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:24.347 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-30T11:18:24.347 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:24.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:18:24.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-30T11:18:24.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:24.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:18:24.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-30T11:18:24.350 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 330712481794 2024-03-30T11:18:24.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-137438953490 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-137438953490 2024-03-30T11:18:24.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:18:24.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953490 2024-03-30T11:18:24.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 137438953490' 2024-03-30T11:18:24.635 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 137438953490 2024-03-30T11:18:24.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:18:24.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953490 -lt 137438953490 2024-03-30T11:18:24.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:18:24.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:24.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:25.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:18:25.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:18:25.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:18:25.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:18:25.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:18:25.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:18:25.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:18:25.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:18:25.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:18:25.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:18:25.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:18:25.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:18:25.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:18:25.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:18:25.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:18:25.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:18:25.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:18:25.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:18:25.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:18:25.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:18:25.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:18:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:18:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:18:25.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:18:25.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:18:25.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:18:26.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:18:26.051 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:18:26.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:18:26.051 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:18:26.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:18:26.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:18:26.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:18:26.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:18:26.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:18:26.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:18:26.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:18:26.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:18:26.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:18:26.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:18:26.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:18:26.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:26.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:26.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:18:26.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:18:26.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:18:26.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:18:26.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:18:26.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:18:26.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:18:26.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:18:26.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:18:26.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:18:26.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:18:26.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:18:26.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:18:26.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:18:26.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:18:26.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:18:26.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:18:26.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:18:26.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:18:26.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:26.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:26.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:18:26.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:18:26.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:18:26.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:18:26.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:18:26.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:26.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:26.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:18:26.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:18:26.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:18:26.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:18:26.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:18:26.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-03-30T11:18:26.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:18:26.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:26.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:18:26.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:18:26.614 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:18:26.615 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:18:26.615 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:26.615 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:26.615 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:18:26.615 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:18:26.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:26.698 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:18:26.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:18:26.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:18:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:18:27.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:18:27.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:18:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:18:27.618 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-30T11:18:27.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-30T11:18:27.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:18:27.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:18:27.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:18:27.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:18:27.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a34ce562-07a0-46ff-99c4-690597785623 2024-03-30T11:18:27.626 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 a34ce562-07a0-46ff-99c4-690597785623 2024-03-30T11:18:27.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a34ce562-07a0-46ff-99c4-690597785623' 2024-03-30T11:18:27.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:18:27.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAD9QdmJo2IJhAAtdmPpGpYP388/1hYb6fE4w== 2024-03-30T11:18:27.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAD9QdmJo2IJhAAtdmPpGpYP388/1hYb6fE4w=="}' 2024-03-30T11:18:27.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a34ce562-07a0-46ff-99c4-690597785623 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:18:27.845 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:18:27.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:18:27.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAD9QdmJo2IJhAAtdmPpGpYP388/1hYb6fE4w== --osd-uuid a34ce562-07a0-46ff-99c4-690597785623 2024-03-30T11:18:27.887 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.893+0000 7f364100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:27.896 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.902+0000 7f364100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:27.898 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.903+0000 7f364100d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:27.898 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.904+0000 7f364100d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:18:27.898 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.904+0000 7f364100d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:18:27.898 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.904+0000 7f364100d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:18:27.988 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:27.994+0000 7f364100d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:18:33.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:18:33.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:18:33.580 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:18:33.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:18:33.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:18:33.946 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:18:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:18:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:18:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:18:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:18:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:18:33.976 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:33.982+0000 7f1649612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:33.995 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:34.001+0000 7f1649612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:33.996 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:34.002+0000 7f1649612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:34.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:18:34.231 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:18:34.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:34.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:34.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:34.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:34.525+0000 7f1649612640 -1 Falling back to public interface 2024-03-30T11:18:35.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:35.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:35.512 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:18:35.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:18:35.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:35.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:35.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:36.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:36.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:36.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:18:36.806 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:18:36.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:36.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:36.844 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:36.850+0000 7f1649612640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:18:37.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:38.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:38.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:38.105 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:18:38.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:18:38.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:38.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:38.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:39.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:39.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:39.426 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:18:39.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:18:39.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:39.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:39.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:40.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:40.793 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:18:40.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:40.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:18:40.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:40.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:41.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:42.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:42.085 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:18:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:18:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:42.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:43.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:43.436 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:18:43.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:43.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:18:43.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:43.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:43.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:44.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:44.794 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:18:44.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:44.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:18:44.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:44.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:45.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:46.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:46.087 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:18:46.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:46.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:18:46.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:46.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:46.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:47.382 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:18:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:18:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:47.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:48.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:48.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:48.793 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:18:48.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:18:48.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:48.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:50.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:50.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:50.399 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:18:50.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:18:50.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:50.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:50.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:51.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:51.865 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:18:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:18:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:52.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:53.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:53.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:53.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:18:53.447 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:18:53.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:53.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:54.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:55.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:55.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:55.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:18:55.181 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:18:55.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:55.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:55.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:56.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:56.668 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:18:56.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:56.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:18:56.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:56.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:57.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:18:58.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:18:58.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:18:58.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:18:58.457 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:18:58.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:18:58.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:18:58.863 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3613910964,v1:127.0.0.1:6803/3613910964] [v2:127.0.0.1:6804/3613910964,v1:127.0.0.1:6805/3613910964] exists,up a34ce562-07a0-46ff-99c4-690597785623 2024-03-30T11:18:58.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:18:58.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:18:58.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:18:58.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fecec0eb-4884-4814-9209-b74851252cbe 2024-03-30T11:18:58.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fecec0eb-4884-4814-9209-b74851252cbe' 2024-03-30T11:18:58.871 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 fecec0eb-4884-4814-9209-b74851252cbe 2024-03-30T11:18:58.871 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:18:58.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAi9QdmcW0NNRAAVKHxfc9OzMIc88aI1XAKBg== 2024-03-30T11:18:58.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAi9QdmcW0NNRAAVKHxfc9OzMIc88aI1XAKBg=="}' 2024-03-30T11:18:58.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fecec0eb-4884-4814-9209-b74851252cbe -i td/osd-scrub-repair/1/new.json 2024-03-30T11:18:59.254 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:18:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:18:59.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAi9QdmcW0NNRAAVKHxfc9OzMIc88aI1XAKBg== --osd-uuid fecec0eb-4884-4814-9209-b74851252cbe 2024-03-30T11:18:59.297 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.303+0000 7f900ae1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:59.338 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.344+0000 7f900ae1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.345+0000 7f900ae1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:18:59.340 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.346+0000 7f900ae1e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:18:59.340 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.346+0000 7f900ae1e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:18:59.340 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.346+0000 7f900ae1e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:18:59.413 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:18:59.419+0000 7f900ae1e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:19:05.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:19:05.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:19:05.482 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:19:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:19:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:19:05.852 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:19:05.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:19:05.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:19:05.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:19:05.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:19:05.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:19:05.881 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:05.887+0000 7fac6bf2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:05.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:05.889+0000 7fac6bf2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:05.884 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:05.890+0000 7fac6bf2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:06.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:19:06.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:19:06.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:19:06.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:19:06.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:19:06.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:06.135 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:19:06.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:19:06.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:06.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:06.400 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:06.406+0000 7fac6bf2a640 -1 Falling back to public interface 2024-03-30T11:19:06.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:07.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:07.414 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:19:07.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:07.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:19:07.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:07.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:07.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:08.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:08.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:08.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:19:08.707 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:19:08.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:08.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:08.981 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:08.987+0000 7fac6bf2a640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:19:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:10.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:10.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:10.004 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:19:10.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:19:10.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:10.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:10.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:11.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:11.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:11.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:19:11.297 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:19:11.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:11.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:11.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:12.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:12.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:12.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:19:12.666 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:19:12.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:12.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:13.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:14.140 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:19:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:19:14.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:14.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:14.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:15.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:15.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:15.615 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:19:15.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:19:15.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:15.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:16.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:17.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:17.003 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:19:17.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:17.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:19:17.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:17.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:17.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:18.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:18.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:18.352 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:19:18.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:19:18.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:18.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:18.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:19.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:19.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:19.649 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:19:19.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:19:19.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:19.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:20.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:21.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:21.060 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:19:21.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:21.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:19:21.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:21.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:21.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:22.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:22.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:22.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:19:22.378 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:19:22.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:22.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:23.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:24.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:24.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:24.352 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:19:24.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:19:24.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:24.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:25.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:26.188 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:19:26.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:19:26.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:26.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:26.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:27.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:27.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:27.967 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:19:27.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:19:27.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:27.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:28.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:29.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:29.710 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:19:29.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:29.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:19:29.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:29.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:30.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:31.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:31.388 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:19:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:19:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:31.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:32.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:33.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:33.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:33.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:19:33.125 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:19:33.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:33.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:19:33.414 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1549280549,v1:127.0.0.1:6811/1549280549] [v2:127.0.0.1:6812/1549280549,v1:127.0.0.1:6813/1549280549] exists,up fecec0eb-4884-4814-9209-b74851252cbe 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:19:33.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:19:33.419 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:19:33.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:19:33.421 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:19:33.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b10d4e7a-ae3a-4d0a-85b0-868ab80c19e6 2024-03-30T11:19:33.421 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 b10d4e7a-ae3a-4d0a-85b0-868ab80c19e6 2024-03-30T11:19:33.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b10d4e7a-ae3a-4d0a-85b0-868ab80c19e6' 2024-03-30T11:19:33.422 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:19:33.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBF9Qdm4VJLGhAAV/Zx1WaoahZmdNYuO3czxg== 2024-03-30T11:19:33.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBF9Qdm4VJLGhAAV/Zx1WaoahZmdNYuO3czxg=="}' 2024-03-30T11:19:33.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b10d4e7a-ae3a-4d0a-85b0-868ab80c19e6 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:19:33.799 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:19:33.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:19:33.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBF9Qdm4VJLGhAAV/Zx1WaoahZmdNYuO3czxg== --osd-uuid b10d4e7a-ae3a-4d0a-85b0-868ab80c19e6 2024-03-30T11:19:33.843 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.849+0000 7fcd6cc2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:33.876 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.882+0000 7fcd6cc2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:33.876 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.883+0000 7fcd6cc2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:33.877 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.883+0000 7fcd6cc2a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:19:33.877 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.883+0000 7fcd6cc2a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:19:33.877 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.883+0000 7fcd6cc2a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:19:33.950 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:33.956+0000 7fcd6cc2a640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:19:40.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:19:40.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:19:40.475 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:19:40.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:19:40.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:19:40.845 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:19:40.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:19:40.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:19:40.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:19:40.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:19:40.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:19:40.875 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:40.881+0000 7f09e0d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:40.877 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:40.883+0000 7f09e0d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:40.878 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:40.884+0000 7f09e0d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:19:41.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:19:41.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:19:41.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:19:41.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:19:41.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:19:41.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:41.355 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:19:41.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:19:41.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:41.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:41.402 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:41.408+0000 7f09e0d8c640 -1 Falling back to public interface 2024-03-30T11:19:41.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:42.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:42.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:42.470 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:19:42.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:19:42.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:42.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:42.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:43.788 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:19:43.794+0000 7f09e0d8c640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:19:43.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:43.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:43.792 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:19:43.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:19:43.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:43.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:44.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:45.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:45.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:45.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:19:45.116 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:19:45.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:45.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:45.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:46.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:46.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:46.485 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:19:46.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:19:46.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:46.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:46.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:47.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:47.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:47.779 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:19:47.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:19:47.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:47.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:48.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:49.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:49.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:49.091 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:19:49.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:19:49.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:49.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:49.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:50.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:50.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:50.447 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:19:50.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:19:50.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:50.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:50.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:51.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:51.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:51.986 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:19:51.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:19:51.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:51.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:52.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:53.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:53.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:53.279 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:19:53.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:19:53.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:53.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:53.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:54.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:54.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:54.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:19:54.716 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:19:54.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:54.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:55.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:56.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:56.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:56.095 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:19:56.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:19:56.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:56.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:56.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:57.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:57.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:57.918 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:19:57.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:19:57.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:57.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:19:58.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:19:59.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:19:59.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:19:59.827 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:19:59.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:19:59.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:19:59.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:20:00.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:01.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:01.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:01.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:20:01.361 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:20:01.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:01.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:20:02.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:03.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:03.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:03.009 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:20:03.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:20:03.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:03.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:20:03.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:04.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:04.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:04.461 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:20:04.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:20:04.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:04.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:20:04.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:05.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:05.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:05.894 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:20:05.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:20:05.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:05.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:20:06.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:07.665 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:20:07.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:20:07.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:07.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:20:07.976 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2668052587,v1:127.0.0.1:6819/2668052587] [v2:127.0.0.1:6820/2668052587,v1:127.0.0.1:6821/2668052587] exists,up b10d4e7a-ae3a-4d0a-85b0-868ab80c19e6 2024-03-30T11:20:07.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:20:07.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:20:07.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:20:07.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-30T11:20:07.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:20:08.248 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:20:08.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:20:08.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:20:08.667 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:20:08.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:20:09.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:20:12.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-30T11:20:12.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:20:12.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:20:12.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:20:12.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:20:12.211 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:20:12.211 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:20:12.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:20:12.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:20:12.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:20:12.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:20:12.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:20:12.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:20:12.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:20:12.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:20:12.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:20:12.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:20:12.587 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:12.587 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:20:12.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:20:12.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:12.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:20:12.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-30T11:20:12.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-30T11:20:12.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-30T11:20:12.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:12.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:20:12.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476747 2024-03-30T11:20:12.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476747 2024-03-30T11:20:12.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-68719476747' 2024-03-30T11:20:12.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:12.811 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:20:12.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920773 2024-03-30T11:20:12.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920773 2024-03-30T11:20:12.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-68719476747 2-141733920773' 2024-03-30T11:20:12.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:12.928 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-30T11:20:12.928 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:12.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:20:12.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-30T11:20:12.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:12.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-30T11:20:12.930 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836502 2024-03-30T11:20:12.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-30T11:20:12.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:13.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-30T11:20:13.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:20:14.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:20:14.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:14.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-30T11:20:14.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:14.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476747 2024-03-30T11:20:14.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:14.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:20:14.571 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476747 2024-03-30T11:20:14.571 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:14.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476747 2024-03-30T11:20:14.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476747' 2024-03-30T11:20:14.572 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 68719476747 2024-03-30T11:20:14.573 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:20:14.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476748 -lt 68719476747 2024-03-30T11:20:14.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:14.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920773 2024-03-30T11:20:14.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:14.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:20:14.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920773 2024-03-30T11:20:14.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:14.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920773 2024-03-30T11:20:14.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920773' 2024-03-30T11:20:14.882 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 141733920773 2024-03-30T11:20:14.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:20:15.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920774 -lt 141733920773 2024-03-30T11:20:15.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:20:15.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:15.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:15.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:15.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:15.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:20:15.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:15.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:15.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:16.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:20:16.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:20:16.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:20:16.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-30T11:20:16.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:20:16.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:20:16.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-30T11:20:16.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:20:16.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-30T11:20:16.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:20:16.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:20:17.275 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:20:17.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:20:18.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-30T11:20:18.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-30T11:20:18.833 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 2 allow_ec_overwrites to true 2024-03-30T11:20:18.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:20:18.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:20:18.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:20:18.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:20:18.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:20:18.846 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:20:18.846 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:20:18.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:20:18.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:20:18.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:20:18.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:20:18.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:20:18.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:20:18.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:20:18.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:20:18.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:20:19.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:20:19.324 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:19.324 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:20:19.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:20:19.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:19.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:20:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-30T11:20:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-30T11:20:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-30T11:20:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:19.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:20:19.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476750 2024-03-30T11:20:19.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476750 2024-03-30T11:20:19.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-68719476750' 2024-03-30T11:20:19.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:19.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:20:19.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920776 2024-03-30T11:20:19.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920776 2024-03-30T11:20:19.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-68719476750 2-141733920776' 2024-03-30T11:20:19.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:19.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-30T11:20:19.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:19.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:20:19.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-30T11:20:19.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:19.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-30T11:20:19.651 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836505 2024-03-30T11:20:19.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-30T11:20:19.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:19.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-30T11:20:19.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:19.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476750 2024-03-30T11:20:19.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:19.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:20:19.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476750 2024-03-30T11:20:19.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:19.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476750 2024-03-30T11:20:19.954 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 68719476750 2024-03-30T11:20:19.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-03-30T11:20:19.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:20:20.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476750 -lt 68719476750 2024-03-30T11:20:20.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:20.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920776 2024-03-30T11:20:20.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:20.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:20:20.288 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920776 2024-03-30T11:20:20.288 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:20.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920776 2024-03-30T11:20:20.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920776' 2024-03-30T11:20:20.289 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 141733920776 2024-03-30T11:20:20.289 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:20:20.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920776 -lt 141733920776 2024-03-30T11:20:20.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:20:20.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:20.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:20.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:20.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:21.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:20:21.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:21.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:21.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:21.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:20:21.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:20:21.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:20:21.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:20:21.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-30T11:20:21.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-30T11:20:21.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:20:21.773 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:20:21.774 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:20:22.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-30T11:20:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:20:22.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-30T11:20:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:20:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:20:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:20:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:20:22.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:20:22.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:20:22.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:20:22.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:20:22.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:20:22.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:20:22.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:20:22.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:20:22.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:20:22.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:20:24.653 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:20:24.653 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-30T11:20:24.653 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:20:25.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:20:25.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:20:25.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:20:25.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:20:25.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:20:25.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:20:25.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:20:25.173 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:20:25.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:20:25.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:20:25.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:20:25.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:20:25.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:20:25.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:20:25.203 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:25.209+0000 7f7728c23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:25.205 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:25.211+0000 7f7728c23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:25.206 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:25.213+0000 7f7728c23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:20:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:20:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:20:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:20:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:20:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:20:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:20:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:25.721 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:25.727+0000 7f7728c23640 -1 Falling back to public interface 2024-03-30T11:20:25.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:26.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:26.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:26.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:20:26.744 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:26.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:26.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:27.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:28.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:28.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:28.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:20:28.036 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:20:28.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:28.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:28.124 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:28.130+0000 7f7728c23640 -1 osd.1 57 log_to_monitors true 2024-03-30T11:20:28.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:29.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:29.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:29.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:20:29.331 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:20:29.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:29.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 58 last_clean_interval [16,57) [v2:127.0.0.1:6810/1939834310,v1:127.0.0.1:6811/1939834310] [v2:127.0.0.1:6812/1939834310,v1:127.0.0.1:6813/1939834310] exists,up fecec0eb-4884-4814-9209-b74851252cbe 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:20:29.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:20:29.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:20:29.647 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:20:29.647 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:20:29.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:20:29.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:20:29.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:20:29.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:20:29.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:20:29.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:20:29.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:20:29.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:20:29.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:20:30.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:20:30.098 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:30.098 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:20:30.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:20:30.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:30.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:20:30.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-30T11:20:30.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-30T11:20:30.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-30T11:20:30.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:30.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:20:30.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939650 2024-03-30T11:20:30.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939650 2024-03-30T11:20:30.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-270582939650' 2024-03-30T11:20:30.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:30.451 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:20:30.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920779 2024-03-30T11:20:30.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920779 2024-03-30T11:20:30.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-270582939650 2-141733920779' 2024-03-30T11:20:30.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:30.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-30T11:20:30.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:30.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:20:30.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-30T11:20:30.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:30.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-30T11:20:30.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-30T11:20:30.559 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836508 2024-03-30T11:20:30.560 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:30.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-30T11:20:30.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:20:31.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:20:31.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:32.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-30T11:20:32.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:32.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-270582939650 2024-03-30T11:20:32.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:32.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:20:32.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-270582939650 2024-03-30T11:20:32.220 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:32.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939650 2024-03-30T11:20:32.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-03-30T11:20:32.221 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 270582939650 2024-03-30T11:20:32.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:20:32.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939650 -lt 270582939650 2024-03-30T11:20:32.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:32.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920779 2024-03-30T11:20:32.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:32.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:20:32.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920779 2024-03-30T11:20:32.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:32.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920779 2024-03-30T11:20:32.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920779' 2024-03-30T11:20:32.538 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 141733920779 2024-03-30T11:20:32.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:20:32.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920779 -lt 141733920779 2024-03-30T11:20:32.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:20:32.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:32.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:33.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:33.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:33.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:20:33.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:33.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:33.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:33.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:20:33.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:20:33.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:20:33.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:20:33.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:20:33.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:34.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:20:34.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:34.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:34.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:34.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:20:34.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:20:34.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:20:34.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:20:34.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:20:34.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:20:34.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:20:34.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:20:34.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:20:35.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-30T11:20:35.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-30T11:20:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-30T11:20:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:20:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:20:35.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:35.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:35.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:20:35.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:35.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:35.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:20:35.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:20:36.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-30T11:20:36.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-30T11:20:36.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:20:36.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:20:36.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:20:36.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-30T11:20:30.857781+0000 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:20:30.857781+0000 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:20:36.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:20:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:20:30.857781+0000 '>' 2024-03-30T11:20:30.857781+0000 2024-03-30T11:20:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:20:37.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:20:37.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:20:37.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:20:37.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:20:37.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:20:37.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:20:37.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:20:37.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:20:30.857781+0000 '>' 2024-03-30T11:20:30.857781+0000 2024-03-30T11:20:37.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:20:38.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:20:38.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:20:38.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:20:38.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:20:38.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:20:38.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:20:38.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:20:39.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:20:30.857781+0000 '>' 2024-03-30T11:20:30.857781+0000 2024-03-30T11:20:39.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:20:40.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:20:40.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:20:40.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:20:40.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:20:40.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:20:40.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:20:40.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:20:40.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:20:36.729844+0000 '>' 2024-03-30T11:20:30.857781+0000 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:20:40.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:20:40.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:20:40.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:20:40.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:20:40.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:20:40.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:20:40.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:20:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:20:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:20:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:20:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:40.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:20:41.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-03-30T11:20:41.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-03-30T11:20:41.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-03-30T11:20:41.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:41.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:20:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939653 2024-03-30T11:20:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939653 2024-03-30T11:20:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-270582939653' 2024-03-30T11:20:41.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:41.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:20:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920782 2024-03-30T11:20:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920782 2024-03-30T11:20:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-270582939653 2-141733920782' 2024-03-30T11:20:41.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:41.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-03-30T11:20:41.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:41.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:20:41.387 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-03-30T11:20:41.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-03-30T11:20:41.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-03-30T11:20:41.389 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836511 2024-03-30T11:20:41.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:41.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-03-30T11:20:41.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:20:42.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:20:42.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:43.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-03-30T11:20:43.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:43.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-270582939653 2024-03-30T11:20:43.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:43.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:20:43.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-270582939653 2024-03-30T11:20:43.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:43.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939653 2024-03-30T11:20:43.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 270582939653' 2024-03-30T11:20:43.013 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 270582939653 2024-03-30T11:20:43.014 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:20:43.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939653 -lt 270582939653 2024-03-30T11:20:43.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:43.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920782 2024-03-30T11:20:43.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:43.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:20:43.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920782 2024-03-30T11:20:43.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:43.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920782 2024-03-30T11:20:43.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920782' 2024-03-30T11:20:43.297 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 141733920782 2024-03-30T11:20:43.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:20:43.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920782 -lt 141733920782 2024-03-30T11:20:43.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:20:43.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:43.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:43.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:44.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:20:44.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:44.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:44.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:20:44.607 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:20:44.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-30T11:20:44.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:20:44.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-30T11:20:45.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:20:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:20:45.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:20:45.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:20:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:20:47.263 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:20:47.264 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-30T11:20:47.264 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:20:47.264 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:20:47.264 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:20:47.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:20:47.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:20:47.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:20:47.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:20:47.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:20:47.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:20:47.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:20:47.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-30T11:20:47.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:20:47.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:20:47.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:20:47.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:20:47.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:20:47.563 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:47.569+0000 7f00e261f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:47.566 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:47.572+0000 7f00e261f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:47.567 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:47.573+0000 7f00e261f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:47.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:20:47.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:20:47.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:20:47.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:20:47.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:20:47.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:47.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:20:47.826 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:20:47.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:47.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:48.081 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:48.087+0000 7f00e261f640 -1 Falling back to public interface 2024-03-30T11:20:48.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:49.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:49.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:49.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:20:49.108 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:49.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:49.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:49.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:50.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:50.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:50.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:20:50.398 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:20:50.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:50.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:50.401 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:50.407+0000 7f00e261f640 -1 osd.1 72 log_to_monitors true 2024-03-30T11:20:50.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:20:51.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:20:51.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:20:51.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:20:51.799 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:20:51.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:20:51.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 73 last_clean_interval [63,72) [v2:127.0.0.1:6810/3345477182,v1:127.0.0.1:6811/3345477182] [v2:127.0.0.1:6812/3345477182,v1:127.0.0.1:6813/3345477182] exists,up fecec0eb-4884-4814-9209-b74851252cbe 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:20:52.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:20:52.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:20:52.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:20:52.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:20:52.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:20:52.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:20:52.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:20:52.240 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:20:52.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:20:52.591 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:20:52.591 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:20:52.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:20:52.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:52.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:20:52.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-30T11:20:52.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-30T11:20:52.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-30T11:20:52.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:52.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:20:52.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-30T11:20:52.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-30T11:20:52.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-335007449090' 2024-03-30T11:20:52.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:20:52.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:20:52.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920785 2024-03-30T11:20:52.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920785 2024-03-30T11:20:52.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-335007449090 2-141733920785' 2024-03-30T11:20:52.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:52.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-30T11:20:52.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:52.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:20:52.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-30T11:20:52.979 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:52.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-30T11:20:52.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-30T11:20:52.980 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836514 2024-03-30T11:20:52.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:53.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-03-30T11:20:53.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:20:54.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:20:54.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:20:54.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836514 2024-03-30T11:20:54.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:54.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-335007449090 2024-03-30T11:20:54.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:54.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:20:54.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-335007449090 2024-03-30T11:20:54.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:54.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-30T11:20:54.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2024-03-30T11:20:54.559 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 335007449090 2024-03-30T11:20:54.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:20:54.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449090 -lt 335007449090 2024-03-30T11:20:54.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:20:54.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920785 2024-03-30T11:20:54.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:20:54.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:20:54.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920785 2024-03-30T11:20:54.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:20:54.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920785 2024-03-30T11:20:54.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920785' 2024-03-30T11:20:54.852 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 141733920785 2024-03-30T11:20:54.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:20:55.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920786 -lt 141733920785 2024-03-30T11:20:55.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:20:55.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:55.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:55.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:20:55.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:20:55.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:20:55.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:20:55.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:20:55.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:20:55.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:20:55.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:20:55.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:20:55.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:20:55.770 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:20:55.770 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:20:56.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:20:56.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:20:56.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:20:56.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:20:56.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:20:56.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:20:56.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:20:56.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:20:56.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:20:56.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:20:56.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:20:56.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:20:56.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:20:56.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:20:56.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:20:56.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:20:56.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:20:56.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:20:56.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:20:56.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:20:56.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:20:56.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:20:56.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:20:56.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:20:56.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:20:56.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:20:56.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:20:56.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:20:56.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:20:56.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:56.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:56.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:20:56.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:20:56.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:20:56.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:20:56.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:20:56.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:20:56.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:20:56.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:20:56.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:20:56.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:20:56.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:20:56.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:20:56.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:20:56.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:20:56.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:20:56.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:20:56.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:20:56.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:56.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:56.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:20:56.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:20:56.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:20:56.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:20:56.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:20:56.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:56.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:56.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:20:56.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:20:56.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:20:56.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:20:56.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:20:56.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-03-30T11:20:56.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:20:56.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:20:56.679 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:56.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:20:56.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:20:56.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:20:56.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:20:56.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:20:56.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:56.865 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:20:56.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:20:56.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:20:56.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:20:56.949 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:20:56.950 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:20:56.950 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:56.950 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:56.950 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:20:56.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:20:56.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:20:57.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-30T11:20:57.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:20:57.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:20:57.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:20:57.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:20:57.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:20:57.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:20:57.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:20:57.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:20:57.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:20:57.841 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:20:57.841 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:57.841 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:57.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:57.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:20:57.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:20:57.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:20:57.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:20:57.875 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:20:57.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5578123d-c4c2-471c-b1cf-23cc450fc53d 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5578123d-c4c2-471c-b1cf-23cc450fc53d' 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 5578123d-c4c2-471c-b1cf-23cc450fc53d 2024-03-30T11:20:57.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:20:57.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZ9QdmZYeJNRAA8ZXxRXKB7jF4AYYdyXEfUg== 2024-03-30T11:20:57.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZ9QdmZYeJNRAA8ZXxRXKB7jF4AYYdyXEfUg=="}' 2024-03-30T11:20:57.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5578123d-c4c2-471c-b1cf-23cc450fc53d -i td/osd-scrub-repair/0/new.json 2024-03-30T11:20:58.137 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:20:58.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:20:58.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZ9QdmZYeJNRAA8ZXxRXKB7jF4AYYdyXEfUg== --osd-uuid 5578123d-c4c2-471c-b1cf-23cc450fc53d 2024-03-30T11:20:58.176 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.182+0000 7f1c9d08b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:58.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.194+0000 7f1c9d08b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:58.189 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.196+0000 7f1c9d08b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:20:58.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.196+0000 7f1c9d08b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:20:58.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.196+0000 7f1c9d08b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:20:58.191 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.196+0000 7f1c9d08b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:20:58.280 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:20:58.286+0000 7f1c9d08b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:21:03.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:21:03.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:21:03.768 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:21:03.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:21:03.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:21:04.140 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:21:04.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:21:04.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:21:04.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:21:04.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:21:04.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:21:04.166 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:04.172+0000 7f4ce0d24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:04.175 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:04.182+0000 7f4ce0d24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:04.176 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:04.183+0000 7f4ce0d24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:04.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:21:04.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:21:04.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:21:04.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:21:04.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:21:04.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:04.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:21:04.449 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:21:04.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:04.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:04.707 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:04.714+0000 7f4ce0d24640 -1 Falling back to public interface 2024-03-30T11:21:04.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:05.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:05.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:05.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:21:05.732 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:21:05.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:05.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:06.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:07.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:07.026 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:21:07.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:07.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:21:07.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:07.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:07.043 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:07.049+0000 7f4ce0d24640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:21:07.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:08.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:08.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:08.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:21:08.470 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:21:08.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:08.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:08.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:09.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:09.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:09.791 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:21:09.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:21:09.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:09.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:10.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:11.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:11.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:11.086 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:21:11.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:21:11.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:11.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:11.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:12.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:12.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:12.380 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:21:12.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:21:12.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:12.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:12.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:13.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:13.675 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:21:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:21:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:13.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:13.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:14.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:14.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:14.970 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:21:14.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:21:14.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:14.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:15.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:16.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:16.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:16.264 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:21:16.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:21:16.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:16.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:16.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:17.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:17.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:17.603 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:21:17.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:21:17.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:17.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:17.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:18.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:18.981 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:21:18.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:18.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:21:18.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:18.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:19.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:20.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:20.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:20.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:21:20.603 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:21:20.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:20.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:21.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:22.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:22.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:22.177 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:21:22.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:21:22.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:22.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:23.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:24.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:24.023 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:21:24.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:24.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:21:24.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:24.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:25.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:26.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:26.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:26.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:21:26.053 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:21:26.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:26.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:26.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:27.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:27.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:27.812 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:21:27.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:21:27.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:27.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:28.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:29.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:29.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:29.141 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:21:29.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:21:29.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:29.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:21:29.513 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1523882113,v1:127.0.0.1:6803/1523882113] [v2:127.0.0.1:6804/1523882113,v1:127.0.0.1:6805/1523882113] exists,up 5578123d-c4c2-471c-b1cf-23cc450fc53d 2024-03-30T11:21:29.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:21:29.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:21:29.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:21:29.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:21:29.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-30T11:21:29.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:21:29.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:21:29.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3' 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:21:29.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:21:29.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC59QdmJgk9IBAA/2/Tk0E0rOhhuW/PXFSExA== 2024-03-30T11:21:29.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC59QdmJgk9IBAA/2/Tk0E0rOhhuW/PXFSExA=="}' 2024-03-30T11:21:29.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:21:29.885 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:21:29.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:21:29.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC59QdmJgk9IBAA/2/Tk0E0rOhhuW/PXFSExA== --osd-uuid f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:21:29.925 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:29.931+0000 7f06c4222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:29.978 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:29.984+0000 7f06c4222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:29.979 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:29.985+0000 7f06c4222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:29.979 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:29.985+0000 7f06c4222640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:21:29.980 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:29.985+0000 7f06c4222640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:21:29.980 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:29.985+0000 7f06c4222640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:21:30.053 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:30.059+0000 7f06c4222640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:21:36.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:21:36.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:21:36.102 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:21:36.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:21:36.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:21:36.466 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:21:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:21:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:21:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:21:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:21:36.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:21:36.492 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:36.498+0000 7f2dffc37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:36.494 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:36.500+0000 7f2dffc37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:36.495 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:36.501+0000 7f2dffc37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:21:36.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:21:36.754 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:21:36.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:36.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:37.021 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:37.028+0000 7f2dffc37640 -1 Falling back to public interface 2024-03-30T11:21:37.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:38.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:38.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:38.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:21:38.047 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:21:38.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:38.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:38.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:39.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:39.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:39.340 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:21:39.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:21:39.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:39.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:39.475 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:21:39.481+0000 7f2dffc37640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:21:39.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:40.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:40.685 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:21:40.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:40.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:21:40.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:40.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:40.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:41.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:21:41.999 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:21:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:41.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:42.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:43.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:43.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:43.355 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:21:43.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:21:43.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:43.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:43.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:44.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:44.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:44.716 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:21:44.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:21:44.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:44.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:45.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:46.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:46.059 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:21:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:21:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:46.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:47.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:47.379 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:21:47.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:47.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:21:47.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:47.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:47.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:48.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:48.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:48.711 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:21:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:21:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:49.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:50.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:50.023 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:21:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:21:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:50.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:51.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:51.354 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:21:51.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:51.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:21:51.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:51.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:51.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:52.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:52.651 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:21:52.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:52.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:21:52.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:52.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:53.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:54.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:54.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:54.151 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:21:54.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:21:54.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:54.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:54.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:55.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:55.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:55.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:21:55.923 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:21:55.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:55.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:57.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:57.719 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:21:57.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:57.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:21:57.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:57.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:58.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:21:59.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:21:59.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:21:59.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:21:59.190 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:21:59.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:21:59.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:21:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:00.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:00.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:00.598 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:22:00.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:22:00.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:00.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:22:01.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:02.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:02.131 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:22:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:22:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:02.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:22:02.522 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1019689993,v1:127.0.0.1:6811/1019689993] [v2:127.0.0.1:6812/1019689993,v1:127.0.0.1:6813/1019689993] exists,up f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:22:02.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:22:02.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6c7600f-bef6-449f-ac06-e37b798e22e5 2024-03-30T11:22:02.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a6c7600f-bef6-449f-ac06-e37b798e22e5' 2024-03-30T11:22:02.529 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 a6c7600f-bef6-449f-ac06-e37b798e22e5 2024-03-30T11:22:02.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:22:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDa9Qdm9Wu+IBAAJI/9qKsIYfMb0CwYagOXDQ== 2024-03-30T11:22:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDa9Qdm9Wu+IBAAJI/9qKsIYfMb0CwYagOXDQ=="}' 2024-03-30T11:22:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6c7600f-bef6-449f-ac06-e37b798e22e5 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:22:02.967 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:22:02.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:22:02.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDa9Qdm9Wu+IBAAJI/9qKsIYfMb0CwYagOXDQ== --osd-uuid a6c7600f-bef6-449f-ac06-e37b798e22e5 2024-03-30T11:22:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.013+0000 7fec38668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:03.051 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.057+0000 7fec38668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:03.052 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.058+0000 7fec38668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:03.053 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.059+0000 7fec38668640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:22:03.053 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.059+0000 7fec38668640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:22:03.053 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.059+0000 7fec38668640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:22:03.126 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:03.132+0000 7fec38668640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:22:09.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:22:09.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:22:09.133 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:22:09.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:22:09.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:22:09.506 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:22:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:22:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:22:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:22:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:22:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:22:09.532 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:09.538+0000 7f86b6612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:09.534 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:09.540+0000 7f86b6612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:09.535 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:09.541+0000 7f86b6612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:09.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:22:09.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:22:09.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:22:09.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:22:09.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:22:09.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:09.790 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:22:09.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:22:09.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:09.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:10.063 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:10.070+0000 7f86b6612640 -1 Falling back to public interface 2024-03-30T11:22:10.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:11.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:11.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:11.073 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:22:11.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:22:11.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:11.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:11.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:12.366 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:22:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:22:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:12.450 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:12.456+0000 7f86b6612640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:22:12.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:13.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:13.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:13.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:22:13.663 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:22:13.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:13.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:13.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:14.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:14.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:14.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:22:14.957 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:22:14.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:14.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:15.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:16.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:16.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:16.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:22:16.250 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:22:16.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:16.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:16.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:17.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:17.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:17.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:22:17.544 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:22:17.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:17.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:17.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:18.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:18.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:18.838 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:22:18.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:22:18.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:18.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:19.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:20.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:20.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:20.178 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:22:20.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:22:20.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:20.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:20.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:21.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:21.501 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:22:21.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:21.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:22:21.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:21.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:21.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:22.793 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:22:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:22:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:23.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:24.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:24.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:24.088 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:22:24.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:22:24.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:24.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:24.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:25.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:25.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:25.903 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:22:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:22:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:25.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:26.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:27.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:27.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:27.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:22:27.712 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:22:27.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:27.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:28.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:29.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:29.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:29.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:22:29.288 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:22:29.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:29.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:29.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:30.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:30.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:30.825 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:22:30.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:22:30.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:30.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:31.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:32.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:32.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:32.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:22:32.534 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:22:32.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:32.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:34.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:34.287 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:22:34.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:34.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:22:34.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:34.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:22:34.611 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3922011484,v1:127.0.0.1:6819/3922011484] [v2:127.0.0.1:6820/3922011484,v1:127.0.0.1:6821/3922011484] exists,up a6c7600f-bef6-449f-ac06-e37b798e22e5 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:22:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:22:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:22:34.617 INFO:tasks.workunit.client.0.smithi003.stdout:add osd3 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:22:34.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 50293913-c7f8-412c-8756-90d1475bf41d' 2024-03-30T11:22:34.618 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:22:34.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD69QdmLQMQJhAAjbFHKg4hORJ/Hgbyi5aFTA== 2024-03-30T11:22:34.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD69QdmLQMQJhAAjbFHKg4hORJ/Hgbyi5aFTA=="}' 2024-03-30T11:22:34.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50293913-c7f8-412c-8756-90d1475bf41d -i td/osd-scrub-repair/3/new.json 2024-03-30T11:22:35.105 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:22:35.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-30T11:22:35.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD69QdmLQMQJhAAjbFHKg4hORJ/Hgbyi5aFTA== --osd-uuid 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:22:35.144 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.151+0000 7fdec4215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:35.189 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.195+0000 7fdec4215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:35.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.196+0000 7fdec4215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:35.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.197+0000 7fdec4215640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:22:35.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.197+0000 7fdec4215640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:22:35.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.197+0000 7fdec4215640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:22:35.273 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:35.279+0000 7fdec4215640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-30T11:22:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-30T11:22:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:22:41.228 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd3 key to auth repository 2024-03-30T11:22:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-30T11:22:41.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:22:41.595 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.3 2024-03-30T11:22:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-30T11:22:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:22:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:22:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:22:41.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:22:41.622 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:41.628+0000 7feaa1c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:41.623 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:41.630+0000 7feaa1c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:41.624 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:41.631+0000 7feaa1c48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:22:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-30T11:22:41.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:22:41.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:41.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:42.143 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:42.149+0000 7feaa1c48640 -1 Falling back to public interface 2024-03-30T11:22:42.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:43.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:43.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:43.161 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:22:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:22:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:43.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:44.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:44.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:44.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:22:44.453 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:22:44.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:44.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:44.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:22:44.538+0000 7feaa1c48640 -1 osd.3 0 log_to_monitors true 2024-03-30T11:22:44.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:45.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:45.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:45.749 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:22:45.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:22:45.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:45.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:46.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:47.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:47.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:47.041 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:22:47.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:22:47.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:47.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:47.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:48.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:48.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:48.395 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:22:48.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:22:48.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:48.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:48.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:49.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:49.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:49.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:22:49.737 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:22:49.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:49.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:50.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:51.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:51.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:51.031 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:22:51.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:22:51.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:51.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:51.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:52.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:52.433 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:22:52.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:52.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:22:52.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:52.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:52.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:53.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:53.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:53.726 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:22:53.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:22:53.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:53.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:54.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:55.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:55.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:55.020 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:22:55.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:22:55.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:55.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:55.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:22:56.503 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:22:56.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:56.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:57.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:22:58.052 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:22:58.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:58.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:22:58.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:22:59.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:22:59.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:22:59.694 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:22:59.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:22:59.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:22:59.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:00.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:01.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:01.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:01.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:23:01.153 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:23:01.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:01.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:01.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:02.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:02.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:02.765 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:23:02.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:23:02.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:02.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:03.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:04.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:04.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:04.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:23:04.515 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:23:04.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:04.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:05.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:06.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:06.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:06.040 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:23:06.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:23:06.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:06.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:06.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:07.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:07.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:07.682 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:23:07.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:23:07.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:07.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:08.014 INFO:tasks.workunit.client.0.smithi003.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1661442573,v1:127.0.0.1:6827/1661442573] [v2:127.0.0.1:6828/1661442573,v1:127.0.0.1:6829/1661442573] exists,up 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:23:08.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:23:08.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:23:08.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:23:08.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-30T11:23:08.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:23:08.288 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:23:08.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:23:08.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:23:08.727 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:23:08.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:23:09.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:23:11.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-30T11:23:11.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:23:11.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:23:11.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:23:11.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:23:11.775 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:23:11.775 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:23:11.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:23:11.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:23:11.776 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:23:11.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:23:11.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:23:11.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:23:11.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:23:11.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:23:11.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:12.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:23:12.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-30T11:23:12.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-30T11:23:12.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-30T11:23:12.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:12.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:23:12.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-30T11:23:12.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-30T11:23:12.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2024-03-30T11:23:12.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:12.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:23:12.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-03-30T11:23:12.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-03-30T11:23:12.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509457' 2024-03-30T11:23:12.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:12.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:23:12.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-03-30T11:23:12.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-03-30T11:23:12.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509457 3-85899345926' 2024-03-30T11:23:12.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:12.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-30T11:23:12.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:12.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:23:12.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-30T11:23:12.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:12.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-30T11:23:12.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-30T11:23:12.722 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836502 2024-03-30T11:23:12.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:13.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-30T11:23:13.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:23:14.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:23:14.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:14.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-30T11:23:14.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:14.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-30T11:23:14.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:14.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:23:14.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-30T11:23:14.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:14.309 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672980 2024-03-30T11:23:14.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-30T11:23:14.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-30T11:23:14.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:23:14.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672980 2024-03-30T11:23:14.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:14.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-03-30T11:23:14.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:14.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:23:14.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-03-30T11:23:14.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:14.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-03-30T11:23:14.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-03-30T11:23:14.597 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509457 2024-03-30T11:23:14.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:23:14.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-03-30T11:23:14.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:14.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345926 2024-03-30T11:23:14.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:14.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:23:14.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345926 2024-03-30T11:23:14.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:14.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-03-30T11:23:14.882 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345926 2024-03-30T11:23:14.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-03-30T11:23:14.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:23:15.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345926 2024-03-30T11:23:15.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:23:15.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:15.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:15.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:23:15.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:23:15.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:23:15.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:23:15.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:15.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:23:16.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-30T11:23:16.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:23:16.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:23:17.409 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:23:17.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:23:18.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-30T11:23:18.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:23:18.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:23:18.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:23:18.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:23:18.424 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:23:18.425 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:23:18.425 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:23:18.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:23:18.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:23:18.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:23:18.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:23:18.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:23:18.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:23:18.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:23:18.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:23:18.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:23:18.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:23:18.813 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:18.813 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:18.814 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:23:18.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:23:18.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:18.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:23:18.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-30T11:23:18.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-30T11:23:18.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-30T11:23:18.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:18.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:23:19.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-30T11:23:19.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-30T11:23:19.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672983' 2024-03-30T11:23:19.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:19.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:23:19.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-30T11:23:19.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-30T11:23:19.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-64424509459' 2024-03-30T11:23:19.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:19.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:23:19.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345929 2024-03-30T11:23:19.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345929 2024-03-30T11:23:19.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672983 2-64424509459 3-85899345929' 2024-03-30T11:23:19.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:19.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-30T11:23:19.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:19.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:23:19.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-30T11:23:19.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:19.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-30T11:23:19.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-30T11:23:19.436 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836504 2024-03-30T11:23:19.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:19.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-30T11:23:19.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:23:20.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:23:20.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:21.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836504 2024-03-30T11:23:21.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:21.025 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-30T11:23:21.025 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:21.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:23:21.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-30T11:23:21.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:21.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-30T11:23:21.028 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672983 2024-03-30T11:23:21.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-30T11:23:21.028 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:23:21.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-30T11:23:21.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:21.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-30T11:23:21.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:21.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:23:21.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-30T11:23:21.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:21.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-30T11:23:21.314 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509459 2024-03-30T11:23:21.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-30T11:23:21.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:23:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-03-30T11:23:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:21.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345929 2024-03-30T11:23:21.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:21.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:23:21.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345929 2024-03-30T11:23:21.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:21.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345929 2024-03-30T11:23:21.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-03-30T11:23:21.599 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345929 2024-03-30T11:23:21.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:23:21.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345929 2024-03-30T11:23:21.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:23:21.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:21.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:22.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:23:22.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:23:22.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:23:22.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:23:22.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:23:22.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:23:22.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:23:22.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:23:22.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:23:22.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:23:22.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:22.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:22.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:23:22.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:23:22.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:23:23.245 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:23:23.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:23:23.739 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:23:23.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:23:23.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:23:23.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:23:24.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-30T11:23:24.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:23:24.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:23:24.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:23:24.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:23:24.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-30T11:23:24.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-30T11:23:24.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T11:23:24.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-30T11:23:24.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-30T11:23:24.297 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:23:24.297 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T11:23:24.617 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:23:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:23:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:23:24.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:23:24.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:23:24.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:23:24.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:23:24.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:23:24.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:23:24.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:23:24.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:23:26.681 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:23:27.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:23:27.198 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:23:27.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:23:27.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:23:27.200 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:23:27.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:23:27.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:23:27.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:23:27.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:23:27.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:23:27.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:23:27.227 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:27.234+0000 7fdb0c213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:27.230 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:27.236+0000 7fdb0c213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:27.231 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:27.237+0000 7fdb0c213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:27.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:27.747 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:27.754+0000 7fdb0c213640 -1 Falling back to public interface 2024-03-30T11:23:27.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:28.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:28.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:28.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:23:28.774 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:28.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:28.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:29.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:30.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:30.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:30.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:23:30.066 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:30.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:30.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:30.182 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:30.188+0000 7fdb0c213640 -1 osd.3 38 log_to_monitors true 2024-03-30T11:23:30.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:31.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:31.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:31.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:23:31.401 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:23:31.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:31.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/3449788097,v1:127.0.0.1:6827/3449788097] [v2:127.0.0.1:6828/3449788097,v1:127.0.0.1:6829/3449788097] exists,up 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:23:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:23:31.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:23:31.717 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:23:31.717 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:23:31.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:23:31.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:23:31.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:23:31.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:23:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:23:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:23:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:23:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:23:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:23:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:23:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:23:32.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:23:32.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:32.128 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:23:32.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-30T11:23:32.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-30T11:23:32.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-30T11:23:32.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:32.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:23:32.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-30T11:23:32.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-30T11:23:32.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672986' 2024-03-30T11:23:32.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:32.479 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:23:32.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-30T11:23:32.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-30T11:23:32.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509463' 2024-03-30T11:23:32.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:32.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:23:32.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-30T11:23:32.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-30T11:23:32.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509463 3-184683593730' 2024-03-30T11:23:32.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:32.689 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-30T11:23:32.689 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:32.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:23:32.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-30T11:23:32.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:32.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-30T11:23:32.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-30T11:23:32.691 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836508 2024-03-30T11:23:32.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:32.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-30T11:23:32.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:23:33.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:23:33.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:34.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-03-30T11:23:34.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:34.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-30T11:23:34.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:34.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:23:34.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-30T11:23:34.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:34.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-30T11:23:34.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-30T11:23:34.277 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672986 2024-03-30T11:23:34.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:23:34.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672986 2024-03-30T11:23:34.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:34.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-30T11:23:34.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:34.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:23:34.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-30T11:23:34.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:34.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-30T11:23:34.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-30T11:23:34.612 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509463 2024-03-30T11:23:34.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:23:34.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-30T11:23:34.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:34.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-184683593730 2024-03-30T11:23:34.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:34.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:23:34.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-184683593730 2024-03-30T11:23:34.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-30T11:23:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-03-30T11:23:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 184683593730 2024-03-30T11:23:34.898 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:23:35.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-30T11:23:35.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:23:35.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:35.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:23:35.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:23:35.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:23:35.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:23:35.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:35.813 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:36.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:23:36.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:23:36.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:23:36.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:23:36.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:23:36.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:23:36.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:23:36.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:23:36.543 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:23:36.543 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:36.543 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:36.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:23:36.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:23:36.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:23:36.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:23:36.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:23:36.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:23:36.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:23:36.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:23:37.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:23:37.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:23:37.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:23:37.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:23:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:23:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:23:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:23:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:23:37.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:37.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:23:37.727 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:23:37.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:23:37.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:23:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:17.298594+0000 '>' 2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:38.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:23:39.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:23:39.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:23:39.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:23:39.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:23:39.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:23:39.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:23:39.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:23:39.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:17.298594+0000 '>' 2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:39.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:23:40.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:23:40.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:23:40.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:23:40.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:23:40.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:23:40.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:23:40.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:23:40.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:17.298594+0000 '>' 2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:40.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:23:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:23:41.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:23:41.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:23:41.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:23:41.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:23:41.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:23:41.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:23:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:17.298594+0000 '>' 2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:41.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:23:42.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:23:42.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:23:42.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:23:42.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:23:42.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:23:42.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:23:42.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:23:43.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:38.176948+0000 '>' 2024-03-30T11:23:17.298594+0000 2024-03-30T11:23:43.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:23:43.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:23:43.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:23:43.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:23:43.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:23:43.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:23:44.924 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:23:44.924 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:23:44.924 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:23:44.924 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:23:45.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:23:45.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:23:45.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:23:45.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:23:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:23:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:23:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:23:45.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:23:45.229 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:45.234+0000 7f6e32037640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:45.231 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:45.237+0000 7f6e32037640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:45.232 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:45.238+0000 7f6e32037640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:45.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:23:45.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:23:45.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:45.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:45.750 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:45.756+0000 7f6e32037640 -1 Falling back to public interface 2024-03-30T11:23:45.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:46.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:46.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:46.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:23:46.775 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:46.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:46.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:47.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:48.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:48.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:48.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:23:48.071 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:48.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:48.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:48.254 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:48.260+0000 7f6e32037640 -1 osd.3 45 log_to_monitors true 2024-03-30T11:23:48.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:49.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:49.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:49.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:23:49.366 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:23:49.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:49.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:23:49.770 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/2528671053,v1:127.0.0.1:6827/2528671053] [v2:127.0.0.1:6828/2528671053,v1:127.0.0.1:6829/2528671053] exists,up 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:23:49.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:23:49.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:23:49.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:23:49.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:23:49.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:23:49.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:23:49.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:23:49.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:23:49.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:23:49.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:23:49.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:23:49.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:50.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:23:50.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-03-30T11:23:50.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-03-30T11:23:50.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-03-30T11:23:50.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:50.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:23:50.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672991 2024-03-30T11:23:50.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672991 2024-03-30T11:23:50.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672991' 2024-03-30T11:23:50.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:50.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:23:50.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-30T11:23:50.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-30T11:23:50.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672991 2-64424509468' 2024-03-30T11:23:50.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:23:50.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:23:50.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-03-30T11:23:50.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-03-30T11:23:50.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672991 2-64424509468 3-210453397506' 2024-03-30T11:23:50.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:50.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-03-30T11:23:50.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:50.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:23:50.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-03-30T11:23:50.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:50.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-03-30T11:23:50.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-03-30T11:23:50.613 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836513 2024-03-30T11:23:50.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:50.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-03-30T11:23:50.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:23:51.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:23:51.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:23:52.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-03-30T11:23:52.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:52.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672991 2024-03-30T11:23:52.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:52.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:23:52.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672991 2024-03-30T11:23:52.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:52.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672991 2024-03-30T11:23:52.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-03-30T11:23:52.188 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672991 2024-03-30T11:23:52.188 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:23:52.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672991 2024-03-30T11:23:52.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:52.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-30T11:23:52.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:52.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:23:52.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-30T11:23:52.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:52.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-30T11:23:52.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-30T11:23:52.475 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509468 2024-03-30T11:23:52.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:23:52.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-03-30T11:23:52.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:23:52.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397506 2024-03-30T11:23:52.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:23:52.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:23:52.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397506 2024-03-30T11:23:52.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:23:52.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-03-30T11:23:52.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397506' 2024-03-30T11:23:52.760 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 210453397506 2024-03-30T11:23:52.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:23:53.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-03-30T11:23:53.058 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:23:53.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:53.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:53.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:23:53.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:23:53.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:23:53.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:23:53.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:23:53.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:23:53.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:23:53.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:23:53.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:23:53.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:23:53.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:23:53.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:23:54.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:23:54.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:23:54.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:23:54.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:23:54.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:23:54.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:23:54.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:23:54.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:23:54.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:23:54.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:23:54.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:23:54.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:23:54.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:23:54.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:23:54.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:23:56.187 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:23:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:23:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:23:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:23:56.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:23:56.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:23:56.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:23:56.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:23:56.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:23:56.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:23:56.734 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:56.740+0000 7fdb26cbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:56.736 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:56.742+0000 7fdb26cbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:56.737 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:56.743+0000 7fdb26cbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:23:56.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:56.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:23:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:57.265+0000 7fdb26cbf640 -1 Falling back to public interface 2024-03-30T11:23:57.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:58.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:58.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:58.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:23:58.285 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:23:58.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:58.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:23:58.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:23:59.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:23:59.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:23:59.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:23:59.577 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:23:59.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:23:59.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:23:59.615 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:23:59.621+0000 7fdb26cbf640 -1 osd.1 50 log_to_monitors true 2024-03-30T11:23:59.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:24:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:24:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [10,50) [v2:127.0.0.1:6810/2614848117,v1:127.0.0.1:6811/2614848117] [v2:127.0.0.1:6812/2614848117,v1:127.0.0.1:6813/2614848117] exists,up f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:24:01.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:24:01.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:24:01.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:24:01.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:24:01.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:24:01.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:24:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:24:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:24:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:24:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:24:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:24:01.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:01.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:24:01.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-03-30T11:24:01.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-03-30T11:24:01.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-03-30T11:24:01.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:01.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:24:01.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201282 2024-03-30T11:24:01.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201282 2024-03-30T11:24:01.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-236223201282' 2024-03-30T11:24:01.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:01.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:24:02.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-30T11:24:02.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-30T11:24:02.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-236223201282 2-64424509471' 2024-03-30T11:24:02.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:02.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:24:02.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-03-30T11:24:02.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-03-30T11:24:02.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-236223201282 2-64424509471 3-210453397509' 2024-03-30T11:24:02.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:02.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-03-30T11:24:02.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:02.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:24:02.136 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-03-30T11:24:02.136 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:02.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-03-30T11:24:02.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-03-30T11:24:02.137 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836516 2024-03-30T11:24:02.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:02.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-03-30T11:24:02.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:02.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201282 2024-03-30T11:24:02.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:02.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:24:02.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201282 2024-03-30T11:24:02.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:02.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201282 2024-03-30T11:24:02.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-03-30T11:24:02.421 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 236223201282 2024-03-30T11:24:02.421 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:02.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201282 -lt 236223201282 2024-03-30T11:24:02.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:02.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-30T11:24:02.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:02.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:24:02.704 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-30T11:24:02.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:02.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-30T11:24:02.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-30T11:24:02.706 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509471 2024-03-30T11:24:02.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:24:03.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509471 -lt 64424509471 2024-03-30T11:24:03.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:03.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-210453397509 2024-03-30T11:24:03.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:03.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:24:03.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-210453397509 2024-03-30T11:24:03.006 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-03-30T11:24:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 210453397509' 2024-03-30T11:24:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 210453397509 2024-03-30T11:24:03.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:03.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397508 -lt 210453397509 2024-03-30T11:24:03.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:24:04.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:24:04.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:04.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397509 2024-03-30T11:24:04.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:24:04.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:04.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:04.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:04.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:05.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:24:05.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:05.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:05.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:24:05.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:05.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:06.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:06.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:24:06.421 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:24:06.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:06.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:24:06.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:06.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:24:06.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:24:06.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:06.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:06.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:06.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:06.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:06.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:06.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:38.176948+0000 '>' 2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:06.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:07.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:07.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:07.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:07.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:07.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:07.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:07.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:08.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:38.176948+0000 '>' 2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:08.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:09.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:09.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:09.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:09.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:09.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:09.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:09.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:09.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:23:38.176948+0000 '>' 2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:09.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:10.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:10.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:10.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:10.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:10.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:10.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:10.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:07.189174+0000 '>' 2024-03-30T11:23:38.176948+0000 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:24:10.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:24:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:24:10.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:24:10.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:24:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:24:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:24:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:24:10.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:24:10.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:24:10.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:24:10.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:24:10.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:24:10.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:24:10.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:24:10.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:24:12.286 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:24:12.286 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:24:12.286 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:24:12.286 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:24:12.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:24:12.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:24:12.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:24:12.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:24:12.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:24:12.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:24:12.556 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:24:12.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:24:12.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:24:12.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:24:12.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:24:12.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:24:12.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:24:12.583 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:24:12.590+0000 7f48dd668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:12.585 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:24:12.592+0000 7f48dd668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:12.587 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:24:12.593+0000 7f48dd668640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:12.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:24:12.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:24:12.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:24:12.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:24:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:24:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:24:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:24:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:12.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:24:13.102 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:24:13.109+0000 7f48dd668640 -1 Falling back to public interface 2024-03-30T11:24:13.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:24:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:24:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:14.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:24:14.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:15.344 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:24:15.351+0000 7f48dd668640 -1 osd.3 57 log_to_monitors true 2024-03-30T11:24:15.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:15.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:15.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:24:15.436 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:24:15.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:15.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 61 up_thru 55 down_at 58 last_clean_interval [49,57) [v2:127.0.0.1:6826/3708163603,v1:127.0.0.1:6827/3708163603] [v2:127.0.0.1:6828/3708163603,v1:127.0.0.1:6829/3708163603] exists,up 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:24:15.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:24:15.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:24:15.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:24:15.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:24:15.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:24:15.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:24:15.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:24:15.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:24:16.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:24:16.255 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:24:16.255 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:24:16.255 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:24:16.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:24:16.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:16.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:24:16.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-03-30T11:24:16.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-03-30T11:24:16.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-03-30T11:24:16.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:16.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:24:16.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201286 2024-03-30T11:24:16.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201286 2024-03-30T11:24:16.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-236223201286' 2024-03-30T11:24:16.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:16.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:24:16.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509475 2024-03-30T11:24:16.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509475 2024-03-30T11:24:16.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-236223201286 2-64424509475' 2024-03-30T11:24:16.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:16.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:24:16.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005058 2024-03-30T11:24:16.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005058 2024-03-30T11:24:16.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-236223201286 2-64424509475 3-261993005058' 2024-03-30T11:24:16.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:16.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-03-30T11:24:16.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:16.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:24:16.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-03-30T11:24:16.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:16.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-03-30T11:24:16.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-03-30T11:24:16.944 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836520 2024-03-30T11:24:16.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:17.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-03-30T11:24:17.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:24:18.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:24:18.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:18.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-03-30T11:24:18.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:18.638 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201286 2024-03-30T11:24:18.638 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:18.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:24:18.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201286 2024-03-30T11:24:18.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:18.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201286 2024-03-30T11:24:18.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201286' 2024-03-30T11:24:18.640 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 236223201286 2024-03-30T11:24:18.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:18.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201286 -lt 236223201286 2024-03-30T11:24:18.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:18.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509475 2024-03-30T11:24:18.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:18.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:24:18.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509475 2024-03-30T11:24:18.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:18.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509475 2024-03-30T11:24:18.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-03-30T11:24:18.923 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509475 2024-03-30T11:24:18.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:24:19.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509475 -lt 64424509475 2024-03-30T11:24:19.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:19.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-261993005058 2024-03-30T11:24:19.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:19.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:24:19.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-261993005058 2024-03-30T11:24:19.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:19.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005058 2024-03-30T11:24:19.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 261993005058' 2024-03-30T11:24:19.209 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 261993005058 2024-03-30T11:24:19.209 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:19.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005058 -lt 261993005058 2024-03-30T11:24:19.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:24:19.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:19.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:19.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:24:19.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:19.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:19.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:19.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:19.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:19.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:19.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:20.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:24:20.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:20.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:20.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:24:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:24:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:24:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:24:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:24:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:24:20.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:24:20.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:20.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:20.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:20.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:20.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:20.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:20.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:20.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:24:20.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:20.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:20.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:21.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:24:21.215 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:24:21.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:24:21.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:24:21.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:24:21.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:24:21.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:21.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:21.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:22.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:24:22.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:22.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:22.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:22.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:24:22.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:24:22.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:24:22.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:24:22.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:24:22.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-30T11:24:22.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:24:22.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:24:22.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165643"' 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165643' 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165645"' 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165645' 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:24:22.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165643 165645' 2024-03-30T11:24:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:24:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:24:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165643 2024-03-30T11:24:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165646: /' 2024-03-30T11:24:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:24:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165648: /' 2024-03-30T11:24:25.413 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.413 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:24:25.413 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:24:25.413 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:24:25.413 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:24:25.413 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:24:25.414 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:24:25.415 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:24:25.416 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:25.417 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:24:25.418 INFO:tasks.workunit.client.0.smithi003.stderr:165646: start osd.1 2024-03-30T11:24:25.419 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:24:25.462 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.462 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:24:25.463 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:24:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:24:25.465 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:24:25.466 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:24:25.467 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: start osd.2 2024-03-30T11:24:25.468 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:24:29.934 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:24:29.934 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:24:29.934 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2024-03-30T11:24:25.452+0000 7f25bdc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2024-03-30T11:24:25.454+0000 7f25bdc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2024-03-30T11:24:25.456+0000 7f25bdc15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 0 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.935 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2024-03-30T11:24:25.974+0000 7f25bdc15640 -1 Falling back to public interface 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 1 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.936 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2024-03-30T11:24:28.683+0000 7f25bdc15640 -1 osd.1 64 log_to_monitors true 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 3 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165646: osd.1 up in weight 1 up_from 69 up_thru 58 down_at 64 last_165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:24:29.937 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2024-03-30T11:24:25.500+0000 7f0a944d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2024-03-30T11:24:25.502+0000 7f0a944d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2024-03-30T11:24:25.503+0000 7f0a944d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 0 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.938 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2024-03-30T11:24:26.025+0000 7f0a944d9640 -1 Falling back to public interface 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 1 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2024-03-30T11:24:29.007+0000 7f0a944d9640 -1 osd.2 63 log_to_monitors true 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 3 2024-03-30T11:24:29.940 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:29.941 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:30.889 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [55,64) [v2:127.0.0.1:6810/3547022475,v1:127.0.0.1:6811/3547022475] [v2:127.0.0.1:6812/3547022475,v1:127.0.0.1:6813/3547022475] exists,up f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:24:30.889 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:24:30.889 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:24:30.889 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:24:30.889 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:24:30.890 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 1 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 2 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: 3' 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:24:30.891 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-03-30T11:24:30.892 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-03-30T11:24:30.892 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-296352743426' 2024-03-30T11:24:30.892 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:31.943 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph s.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 4 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: osd.2 up in weight 1 up_from 70 up_thru 51 down_at 64 last_clean_interval [15,63) [v2:127.0.0.1:6818/230346536,v1:127.0.0.1:6819/230346536] [v2:127.0.0.1:6820/230346536,v1:127.0.0.1:6821/230346536] exists,up a6c7600f-bef6-449f-ac06-e37b798e22e5 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:24:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 1 2024-03-30T11:24:31.945 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 2 2024-03-30T11:24:31.946 INFO:tasks.workunit.client.0.smithi003.stderr:165648: 3' 2024-03-30T11:24:31.946 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:24:31.946 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:31.946 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:24:31.946 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-03-30T11:24:31.947 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-03-30T11:24:31.947 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-03-30T11:24:31.947 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:32.856 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:24:32.856 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743427 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743427 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-296352743427' 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710723 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710723 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-296352743427 2-300647710723' 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005063 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005063 2024-03-30T11:24:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-296352743427 2-300647710723 3-261993005063' 2024-03-30T11:24:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-03-30T11:24:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:24:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-03-30T11:24:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: waiting osd.0 seq 21474836525 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-296352743427 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-296352743427 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743427 2024-03-30T11:24:32.859 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 296352743427' 2024-03-30T11:24:32.860 INFO:tasks.workunit.client.0.smithi003.stderr:165648: waiting osd.1 seq 296352743427 2024-03-30T11:24:32.860 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: ftell osd.2 flush_pg_stats 2024-03-30T11:24:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-03-30T11:24:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-03-30T11:24:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-296352743426 2-300647710722' 2024-03-30T11:24:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005062 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005062 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-296352743426 2-300647710722 3-261993005062' 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: waiting osd.0 seq 21474836524 2024-03-30T11:24:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836524 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-296352743426 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-296352743426 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 296352743426' 2024-03-30T11:24:33.287 INFO:tasks.workunit.client.0.smithi003.stderr:165646: waiting osd.1 seq 296352743426 2024-03-30T11:24:33.288 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:33.288 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743426 -lt 296352743426 2024-03-30T11:24:33.288 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:34.530 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-300647710722 2024-03-30T11:24:34.530 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:34.530 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:24:34.530 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-300647710722 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 300647710722' 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: waiting osd.2 seq 300647710722 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710722 -lt 300647710722 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-261993005062 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:24:34.531 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-261993005062 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005062 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 261993005062' 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: waiting osd.3 seq 261993005062 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005062 -lt 261993005062 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:34.532 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:24:34.533 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:165646: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:165646: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:24:34.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165645 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:lush_pg_stats: test 296352743426 -lt 296352743427 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743427 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-300647710723 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-300647710723 2024-03-30T11:24:35.111 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710723 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 300647710723' 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: waiting osd.2 seq 300647710723 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710723 -lt 300647710723 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-261993005063 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-261993005063 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005063 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 261993005063' 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: waiting osd.3 seq 261993005063 2024-03-30T11:24:35.112 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005063 -lt 261993005063 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:35.113 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:35.742 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:35.742 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:165648: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:24:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:24:35.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:24:35.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:24:35.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:24:35.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:24:35.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:24:36.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:24:36.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:24:36.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:24:36.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:24:36.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:36.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:36.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:36.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:36.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:36.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:24:36.574 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:24:36.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:36.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:36.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:07.189174+0000 '>' 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:36.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:37.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:37.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:37.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:37.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:37.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:37.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:37.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:38.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:07.189174+0000 '>' 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:38.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:39.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:39.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:39.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:39.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:39.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:39.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:39.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:39.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:07.189174+0000 '>' 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:39.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:40.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:40.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:40.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:40.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:40.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:40.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:40.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:40.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:07.189174+0000 '>' 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:40.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:41.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:41.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:41.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:41.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:41.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:41.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:41.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:42.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:07.189174+0000 '>' 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:42.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:24:43.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:24:43.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:24:43.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:24:43.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:24:43.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:24:43.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:24:43.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:24:43.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:36.609579+0000 '>' 2024-03-30T11:24:07.189174+0000 2024-03-30T11:24:43.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168544"' 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168544' 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168546"' 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168546' 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 168544 168546' 2024-03-30T11:24:43.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:24:43.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:24:43.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168544 2024-03-30T11:24:43.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168547: /' 2024-03-30T11:24:43.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:24:43.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168549: /' 2024-03-30T11:24:45.902 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.902 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:24:45.902 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:24:45.902 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:24:45.903 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: _ 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: hinfo_key 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: snapset 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.904 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:24:45.905 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:24:45.906 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:24:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:168547: start osd.1 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:24:45.930 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: _ 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: hinfo_key 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: snapset 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:24:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:24:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:24:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:24:45.935 INFO:tasks.workunit.client.0.smithi003.stderr:168549: start osd.2 2024-03-30T11:24:50.742 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2024-03-30T11:24:45.941+0000 7f18b3c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2024-03-30T11:24:45.944+0000 7f18b3c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2024-03-30T11:24:45.946+0000 7f18b3c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:24:50.743 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 0 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2024-03-30T11:24:46.461+0000 7f18b3c24640 -1 Falling back to public interface 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.744 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 1 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2024-03-30T11:24:49.110+0000 7f18b3c24640 -1 osd.1 72 log_to_monitors true 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.745 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 3 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168547: osd.1 up in weight 1 up_from 77 up_thru 77 down_aovery_ops 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2024-03-30T11:24:45.969+0000 7faea380d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2024-03-30T11:24:45.972+0000 7faea380d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2024-03-30T11:24:45.973+0000 7faea380d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:24:50.746 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 0 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2024-03-30T11:24:46.496+0000 7faea380d640 -1 Falling back to public interface 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 1 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.747 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2024-03-30T11:24:49.161+0000 7faea380d640 -1 osd.2 73 log_to_monitors true 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 3 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:24:50.748 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:24:51.498 INFO:tasks.workunit.client.0.smithi003.stderr:168549: osd.2 up in weight 1 up_from 77 up_thru 51 down_at 73 last_clean_interval [69,72) [v2:127.0.0.1:6810/320521478,v1:127.0.0.1:6811/320521478] [v2:127.0.0.1:6812/320521478,v1:127.0.0.1:6813/320521478] exists,up f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:24:51.499 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 1 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 2 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: 3' 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-03-30T11:24:51.500 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-330712481794' 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: t 73 last_clean_interval [70,73) [v2:127.0.0.1:6818/3318006428,v1:127.0.0.1:6819/3318006428] [v2:127.0.0.1:6820/3318006428,v1:127.0.0.1:6821/3318006428] exists,up a6c7600f-bef6-449f-ac06-e37b798e22e5 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:24:51.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 1 2024-03-30T11:24:51.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 2 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: 3' 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836531 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836531 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531' 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-330712481795' 2024-03-30T11:24:51.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:53.596 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-330712481795 2-330712481795' 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005068 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005068 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836531 1-330712481795 2-330712481795 3-261993005068' 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836531 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:24:53.597 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836531 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836531 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: waiting osd.0 seq 21474836531 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836531 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836531 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481795 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:24:53.598 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481795 2024-03-30T11:24:53.599 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:53.599 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-03-30T11:24:53.599 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481795' 2024-03-30T11:24:53.599 INFO:tasks.workunit.client.0.smithi003.stderr:168549: waiting osd.1 seq 330712481795 2024-03-30T11:24:53.599 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:53.599 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ceph tell osd.2 flush_pg_stats 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-330712481794 2-330712481794' 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005069 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005069 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-330712481794 2-330712481794 3-261993005069' 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-03-30T11:24:53.604 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: waiting osd.0 seq 21474836530 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836530 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:53.605 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:24:53.606 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-30T11:24:53.606 INFO:tasks.workunit.client.0.smithi003.stderr:168547: waiting osd.1 seq 330712481794 2024-03-30T11:24:53.606 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:24:53.606 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-30T11:24:54.944 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481794 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481794 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481794' 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: waiting osd.2 seq 330712481794 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-261993005069 2024-03-30T11:24:54.945 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-261993005069 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005069 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 261993005069' 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: waiting osd.3 seq 261993005069 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005069 -lt 261993005069 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:54.946 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:54.947 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:54.947 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:54.947 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:54.947 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:54.947 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:24:54.947 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/c$seqs 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481795 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481795 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481795' 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: waiting osd.2 seq 330712481795 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:24:54.993 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-261993005068 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-261993005068 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005068 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 261993005068' 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: waiting osd.3 seq 261993005068 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005069 -lt 261993005068 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:54.994 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:24:54.995 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:24:56.501 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:24:56.502 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:24:56.503 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntulone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:56.506 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:24:56.507 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:24:56.508 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:24:56.508 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:24:56.508 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:24:56.508 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:24:57.375 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cep/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:57.376 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:24:57.377 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:24:57.377 INFO:tasks.workunit.client.0.smithi003.stderr:168549: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:24:57.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:24:57.381 INFO:tasks.workunit.client.0.smithi003.stderr:htest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:24:57.381 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:24:57.381 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:24:57.381 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:24:57.381 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:24:57.381 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:24:57.382 INFO:tasks.workunit.client.0.smithi003.stderr:168547: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168546 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:24:57.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:24:57.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:24:57.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171467"' 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171467' 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171469"' 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171469' 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 171467 171469' 2024-03-30T11:24:57.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:24:57.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:24:57.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171467 2024-03-30T11:24:57.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171470: /' 2024-03-30T11:24:57.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:24:57.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171472: /' 2024-03-30T11:25:00.071 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.071 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:25:00.071 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:25:00.071 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.072 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:25:00.073 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:25:00.074 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:00.075 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: start osd.1 2024-03-30T11:25:00.076 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:25:00.318 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.318 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:25:00.318 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:25:00.318 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:25:00.319 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:25:00.320 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:00.321 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:25:00.322 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:25:00.323 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:25:00.323 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:25:00.323 INFO:tasks.workunit.client.0.smithi003.stderr:171470: start osd.3 2024-03-30T11:25:00.323 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2024-03-30T11:25:00.110+0000 7f3a9a637640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2024-03-30T11:25:00.123+0000 7f3a9a637640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2024-03-30T11:25:00.125+0000 7f3a9a637640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:25:04.685 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 0 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2024-03-30T11:25:00.654+0000 7f3a9a637640 -1 Falling back to public interface 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 1 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:04.686 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2024-03-30T11:25:03.315+0000 7f3a9a637640 -1 osd.1 79 log_to_monitors true 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 3 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.687 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:04.879 INFO:tasks.workunit.client.0.smithi003.stderr:171472: osd.1 up in weight 1 up_from 84 up_thru 84 down_at 80 last_171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:25:04.879 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:25:04.879 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:25:04.879 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:25:04.879 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2024-03-30T11:25:00.353+0000 7fe942a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2024-03-30T11:25:00.356+0000 7fe942a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2024-03-30T11:25:00.357+0000 7fe942a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 0 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2024-03-30T11:25:00.872+0000 7fe942a48640 -1 Falling back to public interface 2024-03-30T11:25:04.880 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 1 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:04.881 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2024-03-30T11:25:03.710+0000 7fe942a48640 -1 osd.3 79 log_to_monitors true 2024-03-30T11:25:04.882 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:04.882 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:04.882 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:25:04.882 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 3 2024-03-30T11:25:04.882 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:04.882 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:05.519 INFO:tasks.workunit.client.0.smithi003.stderr:171470: osd.3 up in weight 1 up_from 85 up_thru 85 down_at 80 last_clean_interval [77,79) [v2:127.0.0.1:6810/3093320129,v1:127.0.0.1:6811/3093320129] [v2:127.0.0.1:6812/3093320129,v1:127.0.0.1:6813/3093320129] exists,up f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:25:05.519 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:25:05.520 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 1 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 2 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: 3' 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-03-30T11:25:05.521 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-360777252866' 2024-03-30T11:25:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [61,79) [v2:127.0.0.1:6826/3093949621,v1:127.0.0.1:6827/3093949621] [v2:127.0.0.1:6828/3093949621,v1:127.0.0.1:6829/3093949621] exists,up 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:25:05.653 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 1 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 2 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: 3' 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:25:05.654 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-360777252867' 2024-03-30T11:25:05.655 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:06.623 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481800 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481800 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-360777252867 2-330712481800' 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-360777252867 2-330712481800 3-365072220163' 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-03-30T11:25:06.624 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: waiting osd.0 seq 21474836536 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252867 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252867 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252867' 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: waiting osd.1 seq 360777252867 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252867 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481800 2024-03-30T11:25:06.625 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:06.626 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:25:06.626 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481800 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtell osd.2 flush_pg_stats 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481799 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481799 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-360777252866 2-330712481799' 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-360777252866 2-330712481799 3-365072220162' 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-03-30T11:25:07.811 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: waiting osd.0 seq 21474836535 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836535 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836535 2024-03-30T11:25:07.812 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252866 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252866 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252866' 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: waiting osd.1 seq 360777252866 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-03-30T11:25:07.813 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:1714test/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481800 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481800' 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: waiting osd.2 seq 330712481800 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481800 -lt 330712481800 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:08.201 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: waiting osd.3 seq 365072220163 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:08.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:25:08.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalon72: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481799 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481799 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481799 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481799' 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: waiting osd.2 seq 330712481799 2024-03-30T11:25:09.060 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481800 -lt 330712481799 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220162 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220162 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220162' 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: waiting osd.3 seq 365072220162 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220162 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:25:09.061 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:25:09.062 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:09.447 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.ce/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:25:09.447 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:25:09.447 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:25:09.447 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:25:09.447 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:25:09.447 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:25:09.448 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:25:09.449 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:25:10.558 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:25:10.559 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:10.560 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:10.561 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:10.599 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | notlient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:10.599 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:10.599 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:25:10.600 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-30T11:25:10.601 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-30T11:25:11.202 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.)' 2024-03-30T11:25:11.202 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:171470: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:25:11.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171469 2024-03-30T11:25:11.347 INFO:tasks.workunit.client.0.smithi003.stderr:client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-30T11:25:11.347 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:25:11.347 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:25:11.348 INFO:tasks.workunit.client.0.smithi003.stderr:171472: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:25:11.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:25:11.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:25:11.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:24:36.609579+0000 2024-03-30T11:25:11.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:25:12.178 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:24:36.609579+0000 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:24:36.609579+0000 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:25:12.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:25:12.191 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:25:12.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:36.609579+0000 '>' 2024-03-30T11:24:36.609579+0000 2024-03-30T11:25:12.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:25:13.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:25:13.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:25:13.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:25:13.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:25:13.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:25:13.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:25:13.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:25:13.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:24:36.609579+0000 '>' 2024-03-30T11:24:36.609579+0000 2024-03-30T11:25:13.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:25:14.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:25:14.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:25:14.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:25:14.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:25:14.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:25:14.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:25:14.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:25:12.825417+0000 '>' 2024-03-30T11:24:36.609579+0000 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174563"' 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174563' 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:25:15.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174565"' 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174565' 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 174563 174565' 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174563 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174566: /' 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:25:15.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174568: /' 2024-03-30T11:25:17.532 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:25:17.532 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:25:17.532 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:25:17.532 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:25:17.532 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:25:17.532 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:25:17.533 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: _ 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: hinfo_key 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: snapset 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:25:17.534 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:25:17.535 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:25:17.535 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:25:17.535 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:25:17.535 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:25:17.535 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:25:17.536 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:25:17.537 INFO:tasks.workunit.client.0.smithi003.stderr:174566: start osd.3 2024-03-30T11:25:18.029 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:25:18.029 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:25:18.029 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:25:18.029 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:25:18.030 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: _ 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: hinfo_key 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: snapset 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:25:18.031 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:25:18.032 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:25:18.033 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:25:18.034 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:25:18.034 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:25:18.034 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:25:18.034 INFO:tasks.workunit.client.0.smithi003.stderr:174568: start osd.1 2024-03-30T11:25:21.775 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2024-03-30T11:25:17.571+0000 7f5ea70f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2024-03-30T11:25:17.573+0000 7f5ea70f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2024-03-30T11:25:17.574+0000 7f5ea70f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:25:21.776 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 0 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2024-03-30T11:25:18.095+0000 7f5ea70f5640 -1 Falling back to public interface 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 1 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:21.777 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2024-03-30T11:25:21.128+0000 7f5ea70f5640 -1 osd.3 87 log_to_monitors true 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 3 2024-03-30T11:25:21.778 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:22.791 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-30T11:25:22.791 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:25:22.791 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2024-03-30T11:25:18.064+0000 7f8161d73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2024-03-30T11:25:18.066+0000 7f8161d73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2024-03-30T11:25:18.067+0000 7f8161d73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 0 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:22.792 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2024-03-30T11:25:18.585+0000 7f8161d73640 -1 Falling back to public interface 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 1 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2 2024-03-30T11:25:22.793 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2024-03-30T11:25:21.456+0000 7f8161d73640 -1 osd.1 87 log_to_monitors true 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 3 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:22.794 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:25:22.960 INFO:tasks.workunit.client.0.smithi003.stderr:174568: osd.1 up in weight 1 up_from 93 up_thru 84 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:25:22.960 INFO:tasks.workunit.client.0.smithi003.stderr:174566: osd.3 up in weight 1 up_from 92 up_thru 85 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6810/1009747754,v1:127.0.0.1:6811/1009747754] [v2:127.0.0.1:6812/1009747754,v1:127.0.0.1:6813/1009747754] exists,up 50293913-c7f8-412c-8756-90d1475bf41d 2024-03-30T11:25:22.960 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:25:22.960 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:25:22.960 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:25:22.960 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:25:22.961 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 1 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 2 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: 3' 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:25:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-30T11:25:22.963 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-30T11:25:22.963 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-399431958530' 2024-03-30T11:25:23.813 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 88 last_clean_interval [84,87) [v2:127.0.0.1:6826/3982602569,v1:127.0.0.1:6827/3982602569] [v2:127.0.0.1:6828/3982602569,v1:127.0.0.1:6829/3982602569] exists,up f9ce23aa-5f8c-4f6d-b077-c4c4a8e454e3 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:25:23.814 INFO:tasks.workunit.client.0.smithi003.stderr:174568: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 1 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 2 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: 3' 2024-03-30T11:25:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-399431958531' 2024-03-30T11:25:23.816 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:24.804 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481805 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481805 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-399431958531 2-330712481805' 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-399431958531 2-330712481805 3-395136991235' 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:24.805 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: waiting osd.0 seq 21474836541 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958531 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958531 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:24.806 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958531' 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: waiting osd.1 seq 399431958531 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481805 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:25:24.807 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481805 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/un $ids 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481804 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481804 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-399431958530 2-330712481804' 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-399431958530 2-330712481804 3-395136991234' 2024-03-30T11:25:25.496 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: waiting osd.0 seq 21474836540 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836540 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836540 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:25.497 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-399431958530 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-399431958530 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 399431958530' 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: waiting osd.1 seq 399431958530 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:25:25.498 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-03-30T11:25:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481804 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481804 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481804 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481804' 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: waiting osd.2 seq 330712481804 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481805 -lt 330712481804 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991234 2024-03-30T11:25:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991234 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: waiting osd.3 seq 395136991234 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:26.725 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:26.725 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:26.725 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:26.725 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:25:27.093 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:27.093 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:27.093 INFO:tasks.workunit.client.0.smithi003.stderr:174566: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:25:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:25:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:174566: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:25:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:25:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:25:27.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174565 2024-03-30T11:25:27.429 INFO:tasks.workunit.client.0.smithi003.stderr:buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:27.429 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481805 2024-03-30T11:25:27.429 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481805' 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: waiting osd.2 seq 330712481805 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481805 -lt 330712481805 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: waiting osd.3 seq 395136991235 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:25:27.430 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991235 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991235 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:25:27.431 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:25:27.432 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:25:27.432 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:25:27.782 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:25:27.782 INFO:tasks.workunit.client.0.smithi003.stderr:174568: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:25:27.782 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:25:27.782 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:25:27.782 INFO:tasks.workunit.client.0.smithi003.stderr:174568: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:25:27.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:25:27.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:25:27.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:25:27.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:25:27.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:25:27.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:25:27.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:25:27.942 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:25:27.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:25:27.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:25:27.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:25:27.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:25:27.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:25:27.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:25:27.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:25:27.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:25:27.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:25:27.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:25:27.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:25:27.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:27.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:27.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:25:27.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:25:27.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:25:27.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:25:27.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:25:27.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:25:27.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:25:27.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:25:27.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:25:27.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:25:27.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:25:27.979 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:25:27.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:25:27.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:25:27.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:25:27.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:25:27.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:25:27.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:25:27.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:25:27.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:25:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:25:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:25:27.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:25:27.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:27.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:25:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:25:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:25:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:25:27.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:25:27.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:27.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:27.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-03-30T11:25:27.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:25:27.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:25:28.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:28.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:25:28.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:25:28.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:25:28.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:25:28.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:25:28.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:28.642 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:25:28.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:25:28.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:25:28.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:25:28.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:25:28.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:25:28.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:25:28.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:25:28.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:29.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:29.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:25:29.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:25:29.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:25:29.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:25:29.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:25:29.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:25:29.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a2d1a84-4f6a-4c3a-8e8c-fd2acb946885 2024-03-30T11:25:29.660 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 2a2d1a84-4f6a-4c3a-8e8c-fd2acb946885 2024-03-30T11:25:29.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2a2d1a84-4f6a-4c3a-8e8c-fd2acb946885' 2024-03-30T11:25:29.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:25:29.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCp9gdmUfOuKBAACu+8GHWrB0VlSpNrTgfxwA== 2024-03-30T11:25:29.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCp9gdmUfOuKBAACu+8GHWrB0VlSpNrTgfxwA=="}' 2024-03-30T11:25:29.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a2d1a84-4f6a-4c3a-8e8c-fd2acb946885 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:25:29.880 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:25:29.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:25:29.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCp9gdmUfOuKBAACu+8GHWrB0VlSpNrTgfxwA== --osd-uuid 2a2d1a84-4f6a-4c3a-8e8c-fd2acb946885 2024-03-30T11:25:29.919 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:29.925+0000 7fb710ece640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:29.931 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:29.938+0000 7fb710ece640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:29.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:29.939+0000 7fb710ece640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:29.933 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:29.939+0000 7fb710ece640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:25:29.933 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:29.939+0000 7fb710ece640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:25:29.933 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:29.939+0000 7fb710ece640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:25:30.023 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:30.029+0000 7fb710ece640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:25:35.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:25:35.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:25:35.656 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:25:35.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:25:35.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:25:36.026 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:25:36.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:25:36.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:25:36.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:25:36.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:25:36.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:25:36.053 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:36.059+0000 7f9056311640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:36.072 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:36.079+0000 7f9056311640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:36.074 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:36.080+0000 7f9056311640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:36.349 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:25:36.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:25:36.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:36.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:36.600 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:36.607+0000 7f9056311640 -1 Falling back to public interface 2024-03-30T11:25:36.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:37.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:37.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:37.632 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:25:37.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:25:37.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:37.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:37.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:38.923 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:25:38.929+0000 7f9056311640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:25:38.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:38.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:38.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:25:38.929 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:25:38.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:38.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:40.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:40.367 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:25:40.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:40.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:25:40.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:40.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:40.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:41.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:41.691 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:25:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:25:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:41.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:41.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:42.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:42.985 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:25:42.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:42.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:25:42.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:42.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:43.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:44.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:44.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:44.281 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:25:44.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:25:44.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:44.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:45.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:45.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:45.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:25:45.576 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:25:45.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:45.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:45.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:46.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:46.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:46.871 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:25:46.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:25:46.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:46.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:47.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:48.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:48.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:48.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:25:48.165 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:25:48.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:48.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:48.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:49.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:49.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:49.495 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:25:49.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:25:49.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:49.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:50.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:50.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:50.831 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:25:50.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:25:50.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:50.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:51.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:52.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:52.528 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:25:52.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:52.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:25:52.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:52.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:53.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:54.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:54.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:54.046 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:25:54.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:25:54.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:54.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:54.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:55.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:55.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:55.765 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:25:55.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:25:55.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:55.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:56.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:57.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:57.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:57.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:25:57.778 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:25:57.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:57.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:58.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:25:59.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:25:59.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:25:59.479 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:25:59.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:25:59.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:25:59.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:25:59.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:00.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:00.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:00.916 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:26:00.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:26:00.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:00.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:26:01.210 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1318589912,v1:127.0.0.1:6803/1318589912] [v2:127.0.0.1:6804/1318589912,v1:127.0.0.1:6805/1318589912] exists,up 2a2d1a84-4f6a-4c3a-8e8c-fd2acb946885 2024-03-30T11:26:01.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:26:01.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:26:01.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9a426c65-41a3-47f9-900b-32f84e004811' 2024-03-30T11:26:01.217 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:26:01.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:26:01.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJ9gdmJxc1DhAA4Turb0ZQyDbiQYDlrfbWeg== 2024-03-30T11:26:01.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJ9gdmJxc1DhAA4Turb0ZQyDbiQYDlrfbWeg=="}' 2024-03-30T11:26:01.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a426c65-41a3-47f9-900b-32f84e004811 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:26:01.583 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:26:01.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:26:01.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJ9gdmJxc1DhAA4Turb0ZQyDbiQYDlrfbWeg== --osd-uuid 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:26:01.622 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.629+0000 7f77bd61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:01.667 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.674+0000 7f77bd61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.675+0000 7f77bd61e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.675+0000 7f77bd61e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:26:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.675+0000 7f77bd61e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:26:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.675+0000 7f77bd61e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:26:01.734 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:01.741+0000 7f77bd61e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:26:07.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:26:07.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:26:07.740 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:26:07.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:26:07.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:26:08.107 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:26:08.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:26:08.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:26:08.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:26:08.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:26:08.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:26:08.132 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:08.138+0000 7f5056e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:08.134 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:08.140+0000 7f5056e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:08.135 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:08.141+0000 7f5056e11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:08.389 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:26:08.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:26:08.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:08.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:08.652 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:08.658+0000 7f5056e11640 -1 Falling back to public interface 2024-03-30T11:26:08.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:09.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:09.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:09.670 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:26:09.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:26:09.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:09.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:10.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:10.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:10.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:26:10.963 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:26:10.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:10.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:11.100 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:11.105+0000 7f5056e11640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:26:11.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:12.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:12.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:12.300 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:26:12.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:26:12.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:12.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:13.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:13.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:13.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:26:13.593 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:26:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:13.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:13.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:14.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:14.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:14.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:26:14.905 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:26:14.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:14.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:15.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:16.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:16.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:16.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:26:16.199 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:26:16.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:16.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:16.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:17.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:17.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:17.495 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:26:17.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:26:17.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:17.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:17.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:18.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:18.789 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:26:18.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:18.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:26:18.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:18.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:19.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:20.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:20.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:20.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:26:20.111 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:26:20.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:20.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:20.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:21.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:21.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:21.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:26:21.402 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:26:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:21.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:22.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:22.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:22.699 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:26:22.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:26:22.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:22.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:23.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:24.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:24.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:24.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:26:24.305 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:26:24.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:24.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:24.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:25.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:25.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:25.910 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:26:25.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:26:25.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:25.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:26.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:27.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:27.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:27.637 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:26:27.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:26:27.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:27.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:27.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:28.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:28.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:28.949 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:26:28.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:26:28.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:28.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:29.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:30.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:30.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:30.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:26:30.410 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:26:30.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:30.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:31.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:32.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:32.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:32.155 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:26:32.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:26:32.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:32.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:32.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:33.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:33.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:33.779 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:26:33.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:26:33.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:33.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:26:34.096 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/32819553,v1:127.0.0.1:6811/32819553] [v2:127.0.0.1:6812/32819553,v1:127.0.0.1:6813/32819553] exists,up 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:26:34.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:26:34.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:26:34.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:26:34.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:26:34.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:26:34.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:26:34.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8e7481c-53f2-4033-86f2-f9b2f3a42194 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f8e7481c-53f2-4033-86f2-f9b2f3a42194' 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 f8e7481c-53f2-4033-86f2-f9b2f3a42194 2024-03-30T11:26:34.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:26:34.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDq9gdmulVpBxAAPi3SCVZ2x6VBS1dzlSW7Mg== 2024-03-30T11:26:34.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDq9gdmulVpBxAAPi3SCVZ2x6VBS1dzlSW7Mg=="}' 2024-03-30T11:26:34.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8e7481c-53f2-4033-86f2-f9b2f3a42194 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:26:34.477 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:26:34.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:26:34.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDq9gdmulVpBxAAPi3SCVZ2x6VBS1dzlSW7Mg== --osd-uuid f8e7481c-53f2-4033-86f2-f9b2f3a42194 2024-03-30T11:26:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.523+0000 7f3c94012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:34.562 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.569+0000 7f3c94012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:34.563 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.570+0000 7f3c94012640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:34.564 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.570+0000 7f3c94012640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:26:34.564 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.570+0000 7f3c94012640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:26:34.564 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.570+0000 7f3c94012640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:26:34.637 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:34.644+0000 7f3c94012640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:26:40.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:26:40.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:26:40.641 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:26:40.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:26:40.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:26:41.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:26:41.021 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:26:41.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:26:41.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:26:41.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:26:41.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:26:41.046 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:41.053+0000 7fc4cfc66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:41.048 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:41.055+0000 7fc4cfc66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:41.049 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:41.056+0000 7fc4cfc66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:41.302 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:26:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:26:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:41.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:41.573 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:41.580+0000 7fc4cfc66640 -1 Falling back to public interface 2024-03-30T11:26:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:42.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:42.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:42.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:26:42.585 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:26:42.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:42.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:42.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:43.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:43.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:43.880 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:26:43.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:26:43.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:43.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:43.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:26:43.978+0000 7fc4cfc66640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:26:44.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:45.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:45.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:45.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:26:45.180 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:26:45.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:45.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:45.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:46.474 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:26:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:26:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:46.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:47.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:47.887 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:26:47.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:47.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:26:47.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:47.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:48.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:49.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:49.184 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:26:49.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:49.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:26:49.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:49.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:49.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:50.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:50.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:50.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:26:50.476 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:26:50.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:50.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:50.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:51.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:51.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:51.846 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:26:51.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:26:51.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:51.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:52.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:53.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:53.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:53.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:26:53.199 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:26:53.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:53.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:53.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:54.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:54.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:54.494 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:26:54.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:26:54.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:54.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:54.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:55.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:55.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:55.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:26:55.871 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:26:55.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:55.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:56.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:57.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:57.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:57.525 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:26:57.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:26:57.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:57.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:26:58.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:26:58.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:26:58.994 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:26:58.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:26:58.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:26:58.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:26:59.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:00.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:00.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:00.532 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:27:00.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:27:00.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:00.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:27:01.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:02.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:02.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:02.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:27:02.420 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:27:02.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:02.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:27:03.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:04.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:04.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:04.434 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:27:04.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:27:04.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:04.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:27:05.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:06.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:06.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:06.220 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:27:06.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:27:06.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:06.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:27:06.515 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2650460281,v1:127.0.0.1:6819/2650460281] [v2:127.0.0.1:6820/2650460281,v1:127.0.0.1:6821/2650460281] exists,up f8e7481c-53f2-4033-86f2-f9b2f3a42194 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:27:06.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:27:06.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:27:06.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 32673486-fa38-4822-ad41-0d0ba4c747d4' 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stdout:add osd3 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:27:06.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:27:06.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAK9wdmyTRiIBAAnmhtiyl7Obk5nxXuVC6hwg== 2024-03-30T11:27:06.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAK9wdmyTRiIBAAnmhtiyl7Obk5nxXuVC6hwg=="}' 2024-03-30T11:27:06.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 32673486-fa38-4822-ad41-0d0ba4c747d4 -i td/osd-scrub-repair/3/new.json 2024-03-30T11:27:06.955 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:27:06.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-30T11:27:06.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAK9wdmyTRiIBAAnmhtiyl7Obk5nxXuVC6hwg== --osd-uuid 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:27:06.995 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.002+0000 7f424c971640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:07.057 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.063+0000 7f424c971640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:07.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.064+0000 7f424c971640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:07.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.065+0000 7f424c971640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:27:07.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.065+0000 7f424c971640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:27:07.058 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.065+0000 7f424c971640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:27:07.140 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:07.147+0000 7f424c971640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-30T11:27:12.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-30T11:27:12.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:27:12.872 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd3 key to auth repository 2024-03-30T11:27:12.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-30T11:27:12.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:27:13.237 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.3 2024-03-30T11:27:13.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-30T11:27:13.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:27:13.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:27:13.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:27:13.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:27:13.263 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:13.270+0000 7fde08549640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:13.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:13.272+0000 7fde08549640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:13.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:13.273+0000 7fde08549640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:27:13.519 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:27:13.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:13.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:13.779 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:13.786+0000 7fde08549640 -1 Falling back to public interface 2024-03-30T11:27:13.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:14.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:14.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:14.803 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:27:14.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:27:14.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:14.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:15.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:16.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:16.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:16.096 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:27:16.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:27:16.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:16.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:16.171 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:16.177+0000 7fde08549640 -1 osd.3 0 log_to_monitors true 2024-03-30T11:27:16.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:17.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:17.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:17.389 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:27:17.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:27:17.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:17.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:17.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:18.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:18.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:18.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:27:18.682 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:27:18.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:18.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:18.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:19.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:20.000 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:27:20.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:20.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:27:20.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:20.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:20.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:21.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:21.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:21.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:27:21.315 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:27:21.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:21.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:21.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:22.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:22.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:22.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:27:22.614 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:27:22.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:22.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:22.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:23.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:23.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:23.907 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:27:23.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:27:23.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:23.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:24.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:25.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:25.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:25.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:27:25.315 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:27:25.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:25.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:25.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:26.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:26.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:26.609 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:27:26.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:27:26.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:26.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:26.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:27.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:27.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:27.906 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:27:27.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:27:27.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:27.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:28.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:29.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:29.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:29.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:27:29.612 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:27:29.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:29.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:30.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:31.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:27:31.137 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:27:31.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:31.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:31.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:32.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:32.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:32.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:27:32.827 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:27:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:33.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:34.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:27:34.517 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:27:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:35.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:36.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:36.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:36.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:27:36.395 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:27:36.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:36.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:37.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:38.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:38.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:38.131 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:27:38.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:27:38.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:38.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:38.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:27:39.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:27:39.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:39.569 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:27:39.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:27:39.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:39.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:27:39.859 INFO:tasks.workunit.client.0.smithi003.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4214547719,v1:127.0.0.1:6827/4214547719] [v2:127.0.0.1:6828/4214547719,v1:127.0.0.1:6829/4214547719] exists,up 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:27:39.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:27:39.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:27:39.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:27:39.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-30T11:27:39.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:27:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:27:40.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:27:40.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:27:40.527 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:27:40.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:27:41.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:27:43.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-30T11:27:43.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:27:43.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:27:43.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:27:43.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:27:43.920 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:27:43.920 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:27:43.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:27:43.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:27:43.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:27:44.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:27:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:27:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:27:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:27:44.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:27:44.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:27:44.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:27:44.299 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:27:44.300 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:27:44.300 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:27:44.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:27:44.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:44.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:27:44.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-30T11:27:44.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-30T11:27:44.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-30T11:27:44.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:44.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:27:44.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-30T11:27:44.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-30T11:27:44.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-03-30T11:27:44.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:44.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:27:44.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-30T11:27:44.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-30T11:27:44.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509456' 2024-03-30T11:27:44.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:44.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:27:44.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378631 2024-03-30T11:27:44.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378631 2024-03-30T11:27:44.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509456 3-81604378631' 2024-03-30T11:27:44.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:44.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-30T11:27:44.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:44.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:27:44.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-30T11:27:44.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:44.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-30T11:27:44.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-30T11:27:44.722 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836504 2024-03-30T11:27:44.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:27:45.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836504 2024-03-30T11:27:45.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:27:46.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:27:46.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:27:46.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-30T11:27:46.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:46.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-30T11:27:46.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:46.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:27:46.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-30T11:27:46.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-30T11:27:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-30T11:27:46.341 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672975 2024-03-30T11:27:46.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:27:46.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-03-30T11:27:46.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:46.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-30T11:27:46.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:46.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:27:46.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-30T11:27:46.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:46.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-30T11:27:46.622 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509456 2024-03-30T11:27:46.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-30T11:27:46.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:27:46.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-30T11:27:46.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:46.917 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378631 2024-03-30T11:27:46.917 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:46.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:27:46.918 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378631 2024-03-30T11:27:46.918 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:46.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378631 2024-03-30T11:27:46.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2024-03-30T11:27:46.919 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 81604378631 2024-03-30T11:27:46.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:27:47.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378631 -lt 81604378631 2024-03-30T11:27:47.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:27:47.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:27:47.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:27:47.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:27:47.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:27:47.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:27:47.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:27:47.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:27:47.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:27:47.551 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:27:47.551 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:27:47.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:27:47.828 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:27:47.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:27:47.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:27:48.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-30T11:27:48.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:27:48.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:27:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:27:49.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:27:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-30T11:27:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-30T11:27:50.987 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 2 allow_ec_overwrites to true 2024-03-30T11:27:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:27:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:27:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:27:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:27:51.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:27:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:27:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:27:51.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:27:51.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:27:51.001 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:27:51.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:27:51.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:27:51.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:27:51.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:27:51.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:27:51.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:27:51.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:27:51.379 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:27:51.379 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:27:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:27:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:27:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:51.380 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:27:51.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-30T11:27:51.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-30T11:27:51.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-30T11:27:51.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:27:51.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-30T11:27:51.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-30T11:27:51.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672978' 2024-03-30T11:27:51.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:51.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:27:51.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-30T11:27:51.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-30T11:27:51.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672978 2-64424509459' 2024-03-30T11:27:51.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:27:51.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:27:51.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378633 2024-03-30T11:27:51.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378633 2024-03-30T11:27:51.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672978 2-64424509459 3-81604378633' 2024-03-30T11:27:51.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:51.847 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-30T11:27:51.847 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:51.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:27:51.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-30T11:27:51.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-30T11:27:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-30T11:27:51.849 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836507 2024-03-30T11:27:51.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:27:52.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-30T11:27:52.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:52.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-30T11:27:52.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:52.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:27:52.136 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-30T11:27:52.136 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:52.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-30T11:27:52.137 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672978 2024-03-30T11:27:52.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-30T11:27:52.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:27:52.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-30T11:27:52.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:52.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-30T11:27:52.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:52.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:27:52.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-30T11:27:52.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:52.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-30T11:27:52.421 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509459 2024-03-30T11:27:52.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-30T11:27:52.421 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:27:52.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-30T11:27:52.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:27:52.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-81604378633 2024-03-30T11:27:52.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:27:52.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:27:52.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-81604378633 2024-03-30T11:27:52.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:27:52.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378633 2024-03-30T11:27:52.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2024-03-30T11:27:52.704 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 81604378633 2024-03-30T11:27:52.705 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:27:53.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378632 -lt 81604378633 2024-03-30T11:27:53.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:27:54.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:27:54.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:27:54.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378633 -lt 81604378633 2024-03-30T11:27:54.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:27:54.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:27:54.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:27:54.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:27:54.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:27:54.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:27:54.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:27:54.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:27:54.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:27:55.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:27:55.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-30T11:27:55.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:27:55.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:27:55.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:27:55.679 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:27:55.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:27:56.116 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:27:56.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:27:56.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:27:56.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:27:56.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-30T11:27:56.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:27:56.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:27:56.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:27:56.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:27:56.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-30T11:27:56.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-30T11:27:56.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T11:27:56.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-30T11:27:56.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-30T11:27:56.837 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:27:56.837 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T11:27:57.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-30T11:27:57.121 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:27:57.121 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:27:57.121 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T11:27:57.121 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-30T11:27:57.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-30T11:27:57.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:27:57.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:27:57.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:27:57.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:27:57.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:27:57.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:27:57.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:27:57.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:27:57.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:27:57.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:27:57.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:27:58.975 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:27:59.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:27:59.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:27:59.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:27:59.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:27:59.495 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:27:59.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:27:59.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:27:59.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:27:59.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:27:59.521 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:59.527+0000 7f9db6506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:59.523 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:59.530+0000 7f9db6506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:59.524 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:27:59.531+0000 7f9db6506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:27:59.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:00.042 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:00.049+0000 7f9db6506640 -1 Falling back to public interface 2024-03-30T11:28:00.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:01.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:01.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:01.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:28:01.072 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:01.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:01.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:01.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:02.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:02.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:02.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:28:02.367 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:02.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:02.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:02.391 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:02.397+0000 7f9db6506640 -1 osd.3 37 log_to_monitors true 2024-03-30T11:28:02.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:03.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:03.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:03.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:28:03.785 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:28:03.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:03.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [19,37) [v2:127.0.0.1:6826/1948204883,v1:127.0.0.1:6827/1948204883] [v2:127.0.0.1:6828/1948204883,v1:127.0.0.1:6829/1948204883] exists,up 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:28:04.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:28:04.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:28:04.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:28:04.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:28:04.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:28:04.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:28:04.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:28:04.194 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:04.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:28:04.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-03-30T11:28:04.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-03-30T11:28:04.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-03-30T11:28:04.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:04.600 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:28:04.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-03-30T11:28:04.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-03-30T11:28:04.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672982' 2024-03-30T11:28:04.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:04.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:28:04.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-30T11:28:04.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-30T11:28:04.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672982 2-64424509462' 2024-03-30T11:28:04.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:04.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:28:04.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-03-30T11:28:04.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-03-30T11:28:04.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672982 2-64424509462 3-176093659138' 2024-03-30T11:28:04.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:04.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-03-30T11:28:04.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:04.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:28:04.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-03-30T11:28:04.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:04.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-03-30T11:28:04.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-03-30T11:28:04.941 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836510 2024-03-30T11:28:04.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:05.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836510 2024-03-30T11:28:05.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:28:06.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:28:06.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:06.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-03-30T11:28:06.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:06.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-03-30T11:28:06.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:06.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:28:06.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-03-30T11:28:06.525 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:06.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-03-30T11:28:06.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-03-30T11:28:06.526 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672982 2024-03-30T11:28:06.526 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:28:06.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-03-30T11:28:06.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:06.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-30T11:28:06.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:06.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:28:06.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-30T11:28:06.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:06.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-30T11:28:06.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-30T11:28:06.820 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509462 2024-03-30T11:28:06.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:28:07.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-30T11:28:07.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:07.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-176093659138 2024-03-30T11:28:07.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:07.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:28:07.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-176093659138 2024-03-30T11:28:07.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:07.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-03-30T11:28:07.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-03-30T11:28:07.109 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 176093659138 2024-03-30T11:28:07.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:28:07.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-03-30T11:28:07.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:28:07.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:07.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:07.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:28:07.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:28:07.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:28:07.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:28:07.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:28:07.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:28:07.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:28:07.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:28:08.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:28:08.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:28:08.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:08.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:28:08.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:28:08.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:08.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:08.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:08.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:28:09.202 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:28:09.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:09.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:28:09.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:09.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:28:09.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:28:09.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:09.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:09.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:09.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:09.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:09.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:09.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:27:49.191950+0000 '>' 2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:09.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:10.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:10.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:10.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:10.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:10.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:10.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:10.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:10.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:27:49.191950+0000 '>' 2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:10.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:11.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:11.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:11.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:11.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:11.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:11.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:11.786 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:12.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:27:49.191950+0000 '>' 2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:12.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:13.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:13.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:13.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:13.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:13.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:13.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:13.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:13.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:27:49.191950+0000 '>' 2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:13.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:14.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:14.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:14.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:14.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:14.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:14.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:14.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:14.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:28:09.419273+0000 '>' 2024-03-30T11:27:49.191950+0000 2024-03-30T11:28:14.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:28:14.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:28:14.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:28:14.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:28:14.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:28:15.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:28:15.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:28:15.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:28:15.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:28:15.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:28:15.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:28:15.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:28:15.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:28:16.591 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:28:16.592 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:28:16.592 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:28:16.592 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:28:16.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:28:16.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:28:16.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:28:16.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:28:16.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:28:16.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:28:16.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:28:16.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:28:16.891 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:16.897+0000 7ffb63e2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:16.893 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:16.900+0000 7ffb63e2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:16.894 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:16.901+0000 7ffb63e2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:17.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:28:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:28:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:28:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:28:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:28:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:28:17.160 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:28:17.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:17.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:17.407 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:17.414+0000 7ffb63e2a640 -1 Falling back to public interface 2024-03-30T11:28:17.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:28:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:18.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:19.638 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:19.645+0000 7ffb63e2a640 -1 osd.3 43 log_to_monitors true 2024-03-30T11:28:19.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:19.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:19.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:28:19.734 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:19.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:19.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 47 up_thru 41 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/2307247953,v1:127.0.0.1:6827/2307247953] [v2:127.0.0.1:6828/2307247953,v1:127.0.0.1:6829/2307247953] exists,up 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:28:20.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:28:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:28:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:28:20.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:28:20.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:28:20.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:28:20.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:28:20.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:28:20.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:28:20.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:28:20.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:28:20.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:28:20.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:28:20.619 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:20.619 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:20.619 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:28:20.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:28:20.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:20.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:28:20.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836514 2024-03-30T11:28:20.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836514 2024-03-30T11:28:20.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514' 2024-03-30T11:28:20.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:20.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:28:20.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-03-30T11:28:20.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-03-30T11:28:20.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672986' 2024-03-30T11:28:20.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:20.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:28:21.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509466 2024-03-30T11:28:21.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509466 2024-03-30T11:28:21.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672986 2-64424509466' 2024-03-30T11:28:21.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:21.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:28:21.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462914 2024-03-30T11:28:21.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462914 2024-03-30T11:28:21.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836514 1-42949672986 2-64424509466 3-201863462914' 2024-03-30T11:28:21.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:21.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836514 2024-03-30T11:28:21.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:21.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:28:21.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836514 2024-03-30T11:28:21.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:21.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836514 2024-03-30T11:28:21.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-03-30T11:28:21.326 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836514 2024-03-30T11:28:21.326 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:21.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836514 2024-03-30T11:28:21.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:28:22.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:28:22.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:22.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836515 -lt 21474836514 2024-03-30T11:28:22.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:22.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-03-30T11:28:22.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:22.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:28:22.907 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-03-30T11:28:22.907 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:22.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-03-30T11:28:22.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-03-30T11:28:22.908 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672986 2024-03-30T11:28:22.909 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:28:23.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-03-30T11:28:23.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:23.189 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509466 2024-03-30T11:28:23.189 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:23.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:28:23.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509466 2024-03-30T11:28:23.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:23.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509466 2024-03-30T11:28:23.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-03-30T11:28:23.191 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509466 2024-03-30T11:28:23.191 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:28:23.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509466 2024-03-30T11:28:23.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:23.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462914 2024-03-30T11:28:23.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:23.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:28:23.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462914 2024-03-30T11:28:23.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:23.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462914 2024-03-30T11:28:23.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-03-30T11:28:23.475 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 201863462914 2024-03-30T11:28:23.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:28:23.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462914 -lt 201863462914 2024-03-30T11:28:23.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:28:23.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:23.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:28:24.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:28:24.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:28:24.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:28:24.387 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:24.387 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:24.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:28:24.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:28:24.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:28:24.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:28:24.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:28:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:28:24.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:28:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:28:25.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:28:26.890 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:28:27.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:28:27.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:28:27.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:28:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:28:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:28:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:28:27.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:28:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:28:27.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:28:27.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:28:27.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:28:27.437 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:27.444+0000 7f9f0a573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:27.439 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:27.446+0000 7f9f0a573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:27.441 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:27.448+0000 7f9f0a573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:28:27.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:27.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:28:27.954 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:27.960+0000 7f9f0a573640 -1 Falling back to public interface 2024-03-30T11:28:28.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:29.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:29.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:29.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:28:29.004 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:29.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:29.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:28:29.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:30.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:30.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:30.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:28:30.300 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:30.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:30.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:28:30.366 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:30.372+0000 7f9f0a573640 -1 osd.1 49 log_to_monitors true 2024-03-30T11:28:30.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:31.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:31.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:31.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:28:31.653 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:28:31.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:31.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:28:32.134 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/1897771572,v1:127.0.0.1:6811/1897771572] [v2:127.0.0.1:6812/1897771572,v1:127.0.0.1:6813/1897771572] exists,up 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:28:32.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:28:32.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:28:32.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:28:32.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:28:32.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:28:32.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:28:32.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:28:32.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:28:32.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:28:32.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:28:32.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:28:32.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:28:32.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:28:32.530 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:32.530 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:32.530 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:28:32.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:28:32.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:32.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:28:32.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-03-30T11:28:32.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-03-30T11:28:32.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-03-30T11:28:32.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:32.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:28:32.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233986 2024-03-30T11:28:32.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233986 2024-03-30T11:28:32.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-231928233986' 2024-03-30T11:28:32.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:32.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:28:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509470 2024-03-30T11:28:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509470 2024-03-30T11:28:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-231928233986 2-64424509470' 2024-03-30T11:28:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:32.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462917 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462917 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-231928233986 2-64424509470 3-201863462917' 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:33.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:28:33.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-03-30T11:28:33.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:33.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-03-30T11:28:33.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-03-30T11:28:33.033 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836518 2024-03-30T11:28:33.033 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:33.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836518 2024-03-30T11:28:33.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:28:34.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:28:34.317 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:34.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836518 2024-03-30T11:28:34.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:34.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233986 2024-03-30T11:28:34.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:34.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:28:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233986 2024-03-30T11:28:34.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:34.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233986 2024-03-30T11:28:34.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-03-30T11:28:34.611 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 231928233986 2024-03-30T11:28:34.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:28:34.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233986 -lt 231928233986 2024-03-30T11:28:34.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:34.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509470 2024-03-30T11:28:34.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:34.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:28:34.918 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509470 2024-03-30T11:28:34.918 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:34.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509470 2024-03-30T11:28:34.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-03-30T11:28:34.919 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509470 2024-03-30T11:28:34.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:28:35.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509470 2024-03-30T11:28:35.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:35.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462917 2024-03-30T11:28:35.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:35.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:28:35.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462917 2024-03-30T11:28:35.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:35.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462917 2024-03-30T11:28:35.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462917' 2024-03-30T11:28:35.203 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 201863462917 2024-03-30T11:28:35.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:28:35.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462917 -lt 201863462917 2024-03-30T11:28:35.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:28:35.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:35.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:28:35.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:28:36.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:28:36.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:28:36.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:36.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:28:36.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:36.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:37.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:37.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:28:37.316 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:37.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:37.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:37.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:28:09.419273+0000 '>' 2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:37.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:38.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:38.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:38.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:38.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:38.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:38.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:38.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:38.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:28:09.419273+0000 '>' 2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:38.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:39.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:39.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:39.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:39.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:39.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:39.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:39.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:40.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:28:09.419273+0000 '>' 2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:40.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:41.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:41.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:41.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:41.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:41.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:41.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:28:09.419273+0000 '>' 2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:41.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:28:42.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:28:42.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:28:42.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:28:42.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:28:42.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:28:42.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:28:42.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:28:42.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:28:37.834666+0000 '>' 2024-03-30T11:28:09.419273+0000 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:28:42.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:28:42.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:28:42.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:28:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:28:44.553 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:28:44.554 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:28:44.554 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:28:44.554 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:28:44.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:28:44.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:28:44.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:28:44.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:28:44.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:28:44.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:28:44.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:28:44.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:28:44.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:28:44.828 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:28:44.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:28:44.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:28:44.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:28:44.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:28:44.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:28:44.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:28:44.855 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:44.861+0000 7ff66faf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:44.857 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:44.864+0000 7ff66faf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:44.859 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:44.865+0000 7ff66faf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:28:45.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:28:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:28:45.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:45.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:45.386 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:45.393+0000 7ff66faf5640 -1 Falling back to public interface 2024-03-30T11:28:45.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:46.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:46.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:46.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:28:46.405 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:46.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:46.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:46.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:28:47.572 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:28:47.579+0000 7ff66faf5640 -1 osd.3 56 log_to_monitors true 2024-03-30T11:28:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:28:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:28:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:28:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:28:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:28:48.047 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 60 up_thru 54 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/2967611681,v1:127.0.0.1:6827/2967611681] [v2:127.0.0.1:6828/2967611681,v1:127.0.0.1:6829/2967611681] exists,up 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:28:48.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:28:48.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:28:48.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:28:48.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:28:48.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:28:48.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:28:48.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:28:48.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:28:48.521 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:28:48.521 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:28:48.521 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-03-30T11:28:48.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:28:48.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:48.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:28:48.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836522 2024-03-30T11:28:48.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836522 2024-03-30T11:28:48.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522' 2024-03-30T11:28:48.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:48.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:28:48.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233990 2024-03-30T11:28:48.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233990 2024-03-30T11:28:48.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-231928233990' 2024-03-30T11:28:48.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:48.958 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:28:49.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-03-30T11:28:49.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-03-30T11:28:49.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-231928233990 2-64424509474' 2024-03-30T11:28:49.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:28:49.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:28:49.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-03-30T11:28:49.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-03-30T11:28:49.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-231928233990 2-64424509474 3-257698037762' 2024-03-30T11:28:49.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:49.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836522 2024-03-30T11:28:49.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:49.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:28:49.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836522 2024-03-30T11:28:49.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:49.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836522 2024-03-30T11:28:49.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-03-30T11:28:49.282 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836522 2024-03-30T11:28:49.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:49.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836522 2024-03-30T11:28:49.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:28:50.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:28:50.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:28:50.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836522 2024-03-30T11:28:50.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:50.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233990 2024-03-30T11:28:50.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:50.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:28:50.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233990 2024-03-30T11:28:50.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233990 2024-03-30T11:28:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233990' 2024-03-30T11:28:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 231928233990 2024-03-30T11:28:50.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:28:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233990 -lt 231928233990 2024-03-30T11:28:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:51.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-03-30T11:28:51.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:51.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:28:51.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-03-30T11:28:51.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:51.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-03-30T11:28:51.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-03-30T11:28:51.182 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509474 2024-03-30T11:28:51.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:28:51.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-03-30T11:28:51.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:28:51.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037762 2024-03-30T11:28:51.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:28:51.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:28:51.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037762 2024-03-30T11:28:51.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:28:51.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-03-30T11:28:51.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-03-30T11:28:51.470 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 257698037762 2024-03-30T11:28:51.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:28:51.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-03-30T11:28:51.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:28:51.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:51.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:52.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:28:52.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:28:52.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:28:52.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:28:52.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:28:52.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:28:52.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:28:52.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:28:52.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:28:52.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:28:52.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:28:52.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:28:52.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:28:52.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:28:52.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:28:52.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:28:52.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:28:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191117"' 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191117' 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191119"' 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191119' 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191117 191119' 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191120: /' 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191117 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:28:52.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191122: /' 2024-03-30T11:28:55.740 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.740 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:28:55.741 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:28:55.742 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:28:55.743 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:55.744 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: start osd.1 2024-03-30T11:28:55.745 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:28:55.799 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.799 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:28:55.799 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:28:55.799 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.800 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:28:55.801 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:28:55.802 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:28:55.803 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:28:55.804 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:28:55.804 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:28:55.804 INFO:tasks.workunit.client.0.smithi003.stderr:191122: start osd.2 2024-03-30T11:28:55.804 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2024-03-30T11:28:55.780+0000 7fe5176d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2024-03-30T11:28:55.782+0000 7fe5176d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2024-03-30T11:28:55.783+0000 7fe5176d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:00.407 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 0 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2024-03-30T11:28:56.297+0000 7fe5176d9640 -1 Falling back to public interface 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.408 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 1 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2024-03-30T11:28:59.010+0000 7fe5176d9640 -1 osd.1 62 log_to_monitors true 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.409 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:00.410 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 3 2024-03-30T11:29:00.410 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.410 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191120: osd.1 up in weight 1 up_from 68 up_thru 57 down_at 63 last_191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2024-03-30T11:28:55.836+0000 7fa28641e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2024-03-30T11:28:55.839+0000 7fa28641e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2024-03-30T11:28:55.840+0000 7fa28641e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:00.413 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 0 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2024-03-30T11:28:56.364+0000 7fa28641e640 -1 Falling back to public interface 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 1 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:00.414 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2024-03-30T11:28:59.212+0000 7fa28641e640 -1 osd.2 62 log_to_monitors true 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 3 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:00.415 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:01.207 INFO:tasks.workunit.client.0.smithi003.stderr:191122: osd.2 up in weight 1 up_from 69 up_thru 50 down_at 63 last_clean_interval [15,62) [v2:127.0.0.1:6818/4259712218,v1:127.0.0.1:6819/4259712218] [v2:127.0.0.1:6820/4259712218,v1:127.0.0.1:6821/4259712218] exists,up f8e7481c-53f2-4033-86f2-f9b2f3a42194 2024-03-30T11:29:01.207 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:01.207 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:01.207 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:01.208 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 1 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 2 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: 3' 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-03-30T11:29:01.209 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-292057776130' 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [54,62) [v2:127.0.0.1:6810/3805239994,v1:127.0.0.1:6811/3805239994] [v2:127.0.0.1:6812/3805239994,v1:127.0.0.1:6813/3805239994] exists,up 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:01.210 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 1 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 2 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: 3' 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:01.211 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836526 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836526 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526' 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776131 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776131 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-292057776131' 2024-03-30T11:29:01.212 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743427 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743427 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-292057776131 2-296352743427' 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037765 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037765 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-292057776131 2-296352743427 3-257698037765' 2024-03-30T11:29:02.249 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836526 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836526 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836526 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: waiting osd.0 seq 21474836526 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836526 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776131 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:02.250 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776131 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776131 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: waiting osd.1 seq 292057776131 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776131 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743427 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:02.251 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743427 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtell osd.2 flush_pg_stats 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-292057776130 2-296352743426' 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-292057776130 2-296352743426 3-257698037766' 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-03-30T11:29:02.253 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: waiting osd.0 seq 21474836525 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836525 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776130 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776130 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:02.254 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: waiting osd.1 seq 292057776130 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776130 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743426 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:02.255 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743426 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743427 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: waiting osd.2 seq 296352743427 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743427 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037765 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037765 2024-03-30T11:29:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037765 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037765' 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: waiting osd.3 seq 257698037765 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037765 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:03.833 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-03-30T11:29:03.834 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-03-30T11:29:03.889 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalontest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:03.889 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-03-30T11:29:03.889 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2024-03-30T11:29:03.889 INFO:tasks.workunit.client.0.smithi003.stderr:191122: waiting osd.2 seq 296352743426 2024-03-30T11:29:03.889 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:03.889 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743426 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037766 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037766 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: waiting osd.3 seq 257698037766 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037766 -lt 257698037766 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:03.890 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-03-30T11:29:03.891 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:04.964 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 1 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:29:04.965 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:29:04.966 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs:e/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:05.073 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 1 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:29:05.074 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:29:05.075 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:06.556 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:29:06.557 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:06.558 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:191120: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:06.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191119 2024-03-30T11:29:06.687 INFO:tasks.workunit.client.0.smithi003.stderr: ceph --format json status 2024-03-30T11:29:06.687 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:06.687 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:29:06.687 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:29:06.687 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:29:06.687 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:06.688 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:191122: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:29:06.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:29:06.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:29:06.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:29:06.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:29:06.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:29:06.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:07.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:07.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:28:37.834666+0000 2024-03-30T11:29:07.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:29:07.610 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:29:07.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:28:37.834666+0000 2024-03-30T11:29:07.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:29:07.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:28:37.834666+0000 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:07.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:07.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:29:07.646912+0000 '>' 2024-03-30T11:28:37.834666+0000 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194165"' 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194165' 2024-03-30T11:29:07.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194167"' 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194167' 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 194165 194167' 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194165 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194168: /' 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:29:07.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194170: /' 2024-03-30T11:29:11.093 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.093 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:29:11.093 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:11.094 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: _ 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: hinfo_key 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: snapset 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:29:11.095 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:29:11.096 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:11.097 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:29:11.098 INFO:tasks.workunit.client.0.smithi003.stderr:194170: start osd.2 2024-03-30T11:29:11.169 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.169 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:29:11.169 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:29:11.169 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:29:11.169 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.170 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: _ 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: hinfo_key 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: snapset 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:29:11.171 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:29:11.172 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:11.173 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:11.174 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:29:11.174 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:29:11.174 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:29:11.174 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:29:11.174 INFO:tasks.workunit.client.0.smithi003.stderr:194168: start osd.1 2024-03-30T11:29:15.662 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2024-03-30T11:29:11.132+0000 7f3c75017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2024-03-30T11:29:11.134+0000 7f3c75017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2024-03-30T11:29:11.136+0000 7f3c75017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:15.663 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 0 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2024-03-30T11:29:11.656+0000 7f3c75017640 -1 Falling back to public interface 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 1 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:15.664 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2024-03-30T11:29:14.198+0000 7f3c75017640 -1 osd.2 71 log_to_monitors true 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 3 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.665 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:29:15.860 INFO:tasks.workunit.client.0.smithi003.stderr:194170: osd.2 up in weight 1 up_from 76 up_thru 76 down_aovery_ops 2024-03-30T11:29:15.860 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:29:15.860 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2024-03-30T11:29:11.207+0000 7fb7b8c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2024-03-30T11:29:11.209+0000 7fb7b8c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2024-03-30T11:29:11.210+0000 7fb7b8c0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 0 2024-03-30T11:29:15.861 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2024-03-30T11:29:11.743+0000 7fb7b8c0f640 -1 Falling back to public interface 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 1 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2 2024-03-30T11:29:15.862 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2024-03-30T11:29:14.422+0000 7fb7b8c0f640 -1 osd.1 71 log_to_monitors true 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 3 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:15.863 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194168: osd.1 up in weight 1 up_from 77 up_thru 69 down_at 72 last_clean_interval [69,71) [v2:127.0.0.1:6810/3310192636,v1:127.0.0.1:6811/3310192636] [v2:127.0.0.1:6812/3310192636,v1:127.0.0.1:6813/3310192636] exists,up f8e7481c-53f2-4033-86f2-f9b2f3a42194 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:16.395 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 1 2024-03-30T11:29:16.396 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 2 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: 3' 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-330712481794' 2024-03-30T11:29:16.397 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 72 last_clean_interval [68,71) [v2:127.0.0.1:6818/1548910304,v1:127.0.0.1:6819/1548910304] [v2:127.0.0.1:6820/1548910304,v1:127.0.0.1:6821/1548910304] exists,up 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:16.697 INFO:tasks.workunit.client.0.smithi003.stderr:194168: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:16.698 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 1 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 2 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: 3' 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-330712481795' 2024-03-30T11:29:16.699 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-330712481794 2-326417514498' 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037770 2024-03-30T11:29:18.599 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037770 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-330712481794 2-326417514498 3-257698037770' 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: waiting osd.0 seq 21474836530 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:29:18.600 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836530 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: waiting osd.1 seq 330712481794 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:18.601 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-30T11:29:18.792 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-03-30T11:29:18.792 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-03-30T11:29:18.793 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-03-30T11:29:18.793 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-330712481795 2-326417514499' 2024-03-30T11:29:18.793 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:18.793 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:18.793 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037771 2024-03-30T11:29:18.793 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037771 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-330712481795 2-326417514499 3-257698037771' 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-03-30T11:29:18.794 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:18.795 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-03-30T11:29:18.795 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-03-30T11:29:18.795 INFO:tasks.workunit.client.0.smithi003.stderr:194168: waiting osd.0 seq 21474836532 2024-03-30T11:29:18.795 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:18.795 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836532 2024-03-30T11:29:18.795 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481795 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481795 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481795' 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: waiting osd.1 seq 330712481795 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:18.796 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-03-30T11:29:19.840 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:19.840 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514498 2024-03-30T11:29:19.840 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:19.840 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514498 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514498' 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: waiting osd.2 seq 326417514498 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037770 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037770 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:19.841 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037770 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037770' 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: waiting osd.3 seq 257698037770 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037772 -lt 257698037770 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:19.842 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:19.843 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:19.843 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:20.080 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephte$seqs 2024-03-30T11:29:20.080 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514499 2024-03-30T11:29:20.080 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:20.080 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:20.080 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514499 2024-03-30T11:29:20.080 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514499' 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: waiting osd.2 seq 326417514499 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037771 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037771 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037771 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037771' 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: waiting osd.3 seq 257698037771 2024-03-30T11:29:20.081 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037772 -lt 257698037771 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:20.082 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:194170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:194170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:20.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:20.433 INFO:tasks.workunit.client.0.smithi003.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:20.433 INFO:tasks.workunit.client.0.smithi003.stderr:194168: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:20.433 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:20.433 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:20.433 INFO:tasks.workunit.client.0.smithi003.stderr:194168: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:20.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194167 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:29:20.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:29:20.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:29:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 196846"' 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 196846' 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 196848"' 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 196848' 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/196849: /' 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 196846 196848' 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 196846 2024-03-30T11:29:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:29:20.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/196851: /' 2024-03-30T11:29:23.130 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:23.131 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:29:23.132 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:29:23.133 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:23.133 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:23.134 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: start osd.1 2024-03-30T11:29:23.136 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:29:23.691 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:23.692 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:29:23.693 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:29:23.694 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:29:23.695 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:29:23.696 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:29:23.696 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:29:23.696 INFO:tasks.workunit.client.0.smithi003.stderr:196849: start osd.3 2024-03-30T11:29:23.696 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2024-03-30T11:29:23.170+0000 7efd7359a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2024-03-30T11:29:23.172+0000 7efd7359a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2024-03-30T11:29:23.174+0000 7efd7359a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:27.717 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 0 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2024-03-30T11:29:23.691+0000 7efd7359a640 -1 Falling back to public interface 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:27.718 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 1 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2024-03-30T11:29:26.347+0000 7efd7359a640 -1 osd.1 79 log_to_monitors true 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:27.719 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:27.720 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 3 2024-03-30T11:29:27.720 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:27.720 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196851: osd.1 up in weight 1 up_from 84 up_thru 84 down_at 80 last_196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2024-03-30T11:29:23.727+0000 7f201b210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2024-03-30T11:29:23.729+0000 7f201b210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2024-03-30T11:29:23.730+0000 7f201b210640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:28.205 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 0 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2024-03-30T11:29:24.259+0000 7f201b210640 -1 Falling back to public interface 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:28.206 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 1 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:28.207 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2024-03-30T11:29:27.029+0000 7f201b210640 -1 osd.3 79 log_to_monitors true 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 3 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:28.208 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196849: osd.3 up in weight 1 up_from 86 up_thru 77 down_at 80 last_clean_interval [77,79) [v2:127.0.0.1:6818/1645118520,v1:127.0.0.1:6819/1645118520] [v2:127.0.0.1:6820/1645118520,v1:127.0.0.1:6821/1645118520] exists,up 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:28.674 INFO:tasks.workunit.client.0.smithi003.stderr:196851: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 1 2024-03-30T11:29:28.675 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 2 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: 3' 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-30T11:29:28.676 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-30T11:29:28.677 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-360777252866' 2024-03-30T11:29:28.677 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:29.011 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [60,79) [v2:127.0.0.1:6826/3794118044,v1:127.0.0.1:6827/3794118044] [v2:127.0.0.1:6828/3794118044,v1:127.0.0.1:6829/3794118044] exists,up 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:29:29.011 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:29.011 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:29.012 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 1 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 2 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: 3' 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:29.013 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-03-30T11:29:29.014 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-03-30T11:29:29.014 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-360777252867' 2024-03-30T11:29:29.014 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:30.899 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514502 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514502 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-360777252866 2-326417514502' 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187458 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187458 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-360777252866 2-326417514502 3-369367187458' 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-03-30T11:29:30.900 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: waiting osd.0 seq 21474836535 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836535 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252866 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:30.901 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252866 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252866' 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: waiting osd.1 seq 360777252866 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-03-30T11:29:30.902 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:1968tell osd.2 flush_pg_stats 2024-03-30T11:29:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514503 2024-03-30T11:29:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514503 2024-03-30T11:29:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-360777252867 2-326417514503' 2024-03-30T11:29:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187459 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187459 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-360777252867 2-326417514503 3-369367187459' 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: waiting osd.0 seq 21474836536 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836536 2024-03-30T11:29:31.261 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252867 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252867 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252867' 2024-03-30T11:29:31.262 INFO:tasks.workunit.client.0.smithi003.stderr:196849: waiting osd.1 seq 360777252867 2024-03-30T11:29:31.263 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:31.263 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252867 2024-03-30T11:29:31.263 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514502 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514502 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514502 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514502' 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: waiting osd.2 seq 326417514502 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514504 -lt 326417514502 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:32.161 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187458 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187458 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187458 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187458' 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: waiting osd.3 seq 369367187458 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187458 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:32.162 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:32.163 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:32.507 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.c49: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514503 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514503 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514503 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514503' 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: waiting osd.2 seq 326417514503 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514504 -lt 326417514503 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187459 2024-03-30T11:29:32.508 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187459 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187459 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: waiting osd.3 seq 369367187459 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187459 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:32.509 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:32.510 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:32.510 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:32.510 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:32.510 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:32.510 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:32.520 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:32.521 INFO:tasks.workunit.client.0.smithi003.stderr:196851: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:32.521 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:32.521 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:32.521 INFO:tasks.workunit.client.0.smithi003.stderr:196851: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:32.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:196849: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:196849: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:32.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 196848 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:29:32.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:29:33.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:33.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:33.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:33.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:29:33.708 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:33.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:33.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:33.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:34.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:29:07.646912+0000 '>' 2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:34.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:29:35.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:29:35.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:29:35.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:29:35.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:35.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:35.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:35.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:35.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:29:07.646912+0000 '>' 2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:35.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:29:36.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:29:36.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:29:36.369 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:29:36.369 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:36.369 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:36.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:36.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:36.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:29:07.646912+0000 '>' 2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:36.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:29:37.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:29:37.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:29:37.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:29:37.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:29:37.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:29:37.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:29:37.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:29:34.057249+0000 '>' 2024-03-30T11:29:07.646912+0000 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 199673"' 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 199673' 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:37.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 199675"' 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 199675' 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 199673 199675' 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 199673 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/199676: /' 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:37.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/199678: /' 2024-03-30T11:29:40.514 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:40.515 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: _ 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: hinfo_key 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: snapset 2024-03-30T11:29:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:199676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:29:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:199676: start osd.3 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:29:40.787 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:29:40.788 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: _ 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: hinfo_key 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: snapset 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:40.789 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:40.790 INFO:tasks.workunit.client.0.smithi003.stderr:199678: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:29:40.791 INFO:tasks.workunit.client.0.smithi003.stderr:199678: start osd.1 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2024-03-30T11:29:40.553+0000 7f912ee1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2024-03-30T11:29:40.555+0000 7f912ee1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2024-03-30T11:29:40.556+0000 7f912ee1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:44.735 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 0 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2024-03-30T11:29:41.075+0000 7f912ee1e640 -1 Falling back to public interface 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 1 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2024-03-30T11:29:43.408+0000 7f912ee1e640 -1 osd.3 88 log_to_monitors true 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 3 2024-03-30T11:29:44.737 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:45.274 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2024-03-30T11:29:40.821+0000 7fda3a437640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2024-03-30T11:29:40.830+0000 7fda3a437640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2024-03-30T11:29:40.831+0000 7fda3a437640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:45.275 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 0 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2024-03-30T11:29:41.345+0000 7fda3a437640 -1 Falling back to public interface 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 1 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:45.276 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2024-03-30T11:29:44.095+0000 7fda3a437640 -1 osd.1 88 log_to_monitors true 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 3 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:45.277 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199678: osd.1 up in weight 1 up_from 95 up_thru 95 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: osd.3 up in weight 1 up_from 93 up_thru 93 down_at 89 last_clean_interval [86,88) [v2:127.0.0.1:6818/2164046870,v1:127.0.0.1:6819/2164046870] [v2:127.0.0.1:6820/2164046870,v1:127.0.0.1:6821/2164046870] exists,up 32673486-fa38-4822-ad41-0d0ba4c747d4 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:45.931 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 1 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 2 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: 3' 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-03-30T11:29:45.932 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-03-30T11:29:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-03-30T11:29:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-30T11:29:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-30T11:29:45.933 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-408021893122' 2024-03-30T11:29:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 89 last_clean_interval [84,88) [v2:127.0.0.1:6826/1947583196,v1:127.0.0.1:6827/1947583196] [v2:127.0.0.1:6828/1947583196,v1:127.0.0.1:6829/1947583196] exists,up 9a426c65-41a3-47f9-900b-32f84e004811 2024-03-30T11:29:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:29:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:29:46.059 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:29:46.060 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 1 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 2 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: 3' 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-30T11:29:46.061 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-30T11:29:46.062 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-408021893123' 2024-03-30T11:29:46.062 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stan $ids 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514508 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514508 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-408021893122 2-326417514508' 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:48.194 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-408021893122 2-326417514508 3-399431958530' 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: waiting osd.0 seq 21474836540 2024-03-30T11:29:48.195 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836540 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893122 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893122 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893122' 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: waiting osd.1 seq 408021893122 2024-03-30T11:29:48.196 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:48.197 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 408021893122 2024-03-30T11:29:48.197 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:29:48.197 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:29:48.197 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:48.197 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893122 2024-03-30T11:29:48.334 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/uts: ceph tell osd.2 flush_pg_stats 2024-03-30T11:29:48.334 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514509 2024-03-30T11:29:48.334 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514509 2024-03-30T11:29:48.334 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-408021893123 2-326417514509' 2024-03-30T11:29:48.334 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958531 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958531 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-408021893123 2-326417514509 3-399431958531' 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: waiting osd.0 seq 21474836541 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:48.335 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836541 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836541 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-408021893123 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-408021893123 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 408021893123' 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: waiting osd.1 seq 408021893123 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:29:48.336 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893123 2024-03-30T11:29:49.446 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:49.446 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514508 2024-03-30T11:29:49.446 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514508 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514508 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514508' 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: waiting osd.2 seq 326417514508 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514509 -lt 326417514508 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958530 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958530 2024-03-30T11:29:49.447 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958530' 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: waiting osd.3 seq 399431958530 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958530 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:49.448 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:49.449 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:49.449 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:49.449 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/q$seqs 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-326417514509 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-326417514509 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514509 2024-03-30T11:29:49.631 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 326417514509' 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: waiting osd.2 seq 326417514509 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514509 -lt 326417514509 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-399431958531 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-399431958531 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958531 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 399431958531' 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: waiting osd.3 seq 399431958531 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958531 2024-03-30T11:29:49.632 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:29:49.633 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtea/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:199676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:199676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:49.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:29:49.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 199675 2024-03-30T11:29:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:st/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:29:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:199678: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:29:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:29:50.024 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:199678: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:29:50.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:29:50.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:29:50.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:29:50.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:50.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:50.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:50.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:29:50.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:29:50.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:29:50.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:29:50.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:29:50.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:29:50.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:29:50.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:29:50.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:29:50.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:29:50.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:29:50.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:29:50.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:29:50.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:29:50.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:50.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:50.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:29:50.246 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:29:50.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:29:50.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:29:50.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:29:50.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:29:50.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:29:50.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:29:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:29:50.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:29:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:29:50.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:29:50.250 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:29:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:29:50.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:29:50.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:29:50.251 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:29:50.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:29:50.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:29:50.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:29:50.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:29:50.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:50.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:50.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:29:50.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:29:50.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:29:50.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:29:50.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:29:50.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:50.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:50.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:29:50.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:29:50.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:29:50.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:29:50.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:50.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:29:50.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:29:50.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:50.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:29:50.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:29:50.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:29:50.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:29:50.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:29:50.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:29:50.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:29:50.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:29:50.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:29:51.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:51.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:51.816 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:29:51.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:29:51.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-30T11:29:51.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:29:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:29:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:29:51.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:29:51.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5791c2e-c96e-4e28-ad79-48f27543326f 2024-03-30T11:29:51.851 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 b5791c2e-c96e-4e28-ad79-48f27543326f 2024-03-30T11:29:51.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b5791c2e-c96e-4e28-ad79-48f27543326f' 2024-03-30T11:29:51.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:29:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCv9wdmnVcINBAAH3IvZqh/XRv1Zv0C/fMo3w== 2024-03-30T11:29:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCv9wdmnVcINBAAH3IvZqh/XRv1Zv0C/fMo3w=="}' 2024-03-30T11:29:51.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5791c2e-c96e-4e28-ad79-48f27543326f -i td/osd-scrub-repair/0/new.json 2024-03-30T11:29:52.070 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:29:52.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:29:52.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCv9wdmnVcINBAAH3IvZqh/XRv1Zv0C/fMo3w== --osd-uuid b5791c2e-c96e-4e28-ad79-48f27543326f 2024-03-30T11:29:52.109 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.115+0000 7f1877b0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:52.121 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.128+0000 7f1877b0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:52.122 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.129+0000 7f1877b0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:52.122 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.129+0000 7f1877b0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:29:52.122 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.129+0000 7f1877b0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:29:52.122 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.129+0000 7f1877b0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:29:52.213 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:52.219+0000 7f1877b0d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:29:57.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:29:57.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:29:57.760 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:29:57.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:29:57.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:29:58.154 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:29:58.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:29:58.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:29:58.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:29:58.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:29:58.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:29:58.180 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:58.188+0000 7f9441971640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:58.211 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:58.218+0000 7f9441971640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:58.212 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:58.219+0000 7f9441971640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:29:58.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:29:58.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:58.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:29:58.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:29:58.735 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:29:58.742+0000 7f9441971640 -1 Falling back to public interface 2024-03-30T11:29:59.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:29:59.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:29:59.719 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:29:59.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:29:59.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:29:59.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:00.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:01.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:01.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:01.011 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:30:01.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:30:01.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:01.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:01.052 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:01.058+0000 7f9441971640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:30:01.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:02.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:02.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:02.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:30:02.319 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:30:02.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:02.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:02.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:03.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:03.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:03.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:30:03.622 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:30:03.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:03.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:03.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:04.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:04.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:04.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:30:04.995 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:30:04.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:04.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:05.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:06.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:06.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:06.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:30:06.287 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:30:06.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:06.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:06.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:07.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:07.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:07.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:30:07.640 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:30:07.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:07.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:07.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:08.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:08.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:08.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:30:08.997 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:30:08.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:08.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:09.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:10.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:10.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:10.295 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:30:10.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:30:10.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:10.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:10.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:11.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:11.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:11.596 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:30:11.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:30:11.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:11.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:11.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:12.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:12.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:12.975 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:30:12.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:30:12.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:12.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:13.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:14.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:14.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:14.508 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:30:14.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:30:14.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:14.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:15.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:16.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:16.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:16.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:30:16.408 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:30:16.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:16.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:17.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:18.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:18.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:18.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:30:18.085 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:30:18.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:18.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:18.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:19.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:19.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:19.922 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:30:19.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:30:19.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:19.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:20.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:21.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:21.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:21.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:30:21.299 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:30:21.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:21.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:22.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:23.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:23.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:23.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:30:23.035 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:30:23.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:23.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:30:23.351 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2163420359,v1:127.0.0.1:6803/2163420359] [v2:127.0.0.1:6804/2163420359,v1:127.0.0.1:6805/2163420359] exists,up b5791c2e-c96e-4e28-ad79-48f27543326f 2024-03-30T11:30:23.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:30:23.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:30:23.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:30:23.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:30:23.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:30:23.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:30:23.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:30:23.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:30:23.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:30:23.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:30:23.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:30:23.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=633a7106-2740-40a5-b54e-c48644bba0d2 2024-03-30T11:30:23.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 633a7106-2740-40a5-b54e-c48644bba0d2' 2024-03-30T11:30:23.358 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 633a7106-2740-40a5-b54e-c48644bba0d2 2024-03-30T11:30:23.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:30:23.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDP9wdmQ7ufFhAADIhElNyQFun/tB2nnMrBgA== 2024-03-30T11:30:23.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDP9wdmQ7ufFhAADIhElNyQFun/tB2nnMrBgA=="}' 2024-03-30T11:30:23.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 633a7106-2740-40a5-b54e-c48644bba0d2 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:30:23.744 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:30:23.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:30:23.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDP9wdmQ7ufFhAADIhElNyQFun/tB2nnMrBgA== --osd-uuid 633a7106-2740-40a5-b54e-c48644bba0d2 2024-03-30T11:30:23.784 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.791+0000 7f344180f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:23.820 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.827+0000 7f344180f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:23.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.828+0000 7f344180f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:23.822 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.829+0000 7f344180f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:30:23.822 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.829+0000 7f344180f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:30:23.822 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.829+0000 7f344180f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:30:23.895 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:23.902+0000 7f344180f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:30:30.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:30:30.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:30:30.059 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:30:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:30:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:30:30.430 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:30:30.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:30:30.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:30:30.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:30:30.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:30:30.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:30:30.456 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:30.463+0000 7ffbc2bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:30.457 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:30.465+0000 7ffbc2bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:30.458 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:30.466+0000 7ffbc2bbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:30.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:30:30.716 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:30:30.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:30.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:30.973 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:30.980+0000 7ffbc2bbd640 -1 Falling back to public interface 2024-03-30T11:30:31.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:32.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:32.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:32.034 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:30:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:30:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:32.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:32.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:33.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:33.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:33.330 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:30:33.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:30:33.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:33.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:33.387 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:33.393+0000 7ffbc2bbd640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:30:33.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:34.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:34.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:34.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:30:34.625 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:30:34.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:34.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:34.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:35.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:35.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:35.993 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:30:35.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:30:35.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:35.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:36.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:37.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:37.430 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:30:37.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:37.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:30:37.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:37.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:37.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:38.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:38.722 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:30:38.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:38.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:30:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:39.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:40.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:40.017 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:30:40.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:40.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:30:40.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:40.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:40.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:41.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:41.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:41.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:30:41.437 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:30:41.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:41.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:41.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:42.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:42.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:42.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:30:42.738 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:30:42.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:42.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:43.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:44.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:44.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:44.035 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:30:44.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:30:44.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:44.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:44.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:45.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:45.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:45.480 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:30:45.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:30:45.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:45.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:46.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:47.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:47.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:47.094 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:30:47.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:30:47.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:47.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:47.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:48.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:30:48.847 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:30:48.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:48.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:49.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:50.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:50.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:50.541 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:30:50.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:30:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:50.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:51.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:52.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:52.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:52.252 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:30:52.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:30:52.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:52.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:52.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:53.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:53.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:53.672 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:30:53.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:30:53.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:53.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:54.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:30:55.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:30:55.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:30:55.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:30:55.308 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:30:55.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:30:55.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:30:55.627 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/395541537,v1:127.0.0.1:6811/395541537] [v2:127.0.0.1:6812/395541537,v1:127.0.0.1:6813/395541537] exists,up 633a7106-2740-40a5-b54e-c48644bba0d2 2024-03-30T11:30:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:30:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:30:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:30:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:30:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-30T11:30:55.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:30:55.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:30:55.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:30:55.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:30:55.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcdeba07-682b-453c-868b-832e3c6c50fd 2024-03-30T11:30:55.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dcdeba07-682b-453c-868b-832e3c6c50fd' 2024-03-30T11:30:55.642 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 dcdeba07-682b-453c-868b-832e3c6c50fd 2024-03-30T11:30:55.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:30:55.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDv9wdmKIYOJxAAJ3i8DBQzT8Ul9PkelBZuHg== 2024-03-30T11:30:55.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDv9wdmKIYOJxAAJ3i8DBQzT8Ul9PkelBZuHg=="}' 2024-03-30T11:30:55.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcdeba07-682b-453c-868b-832e3c6c50fd -i td/osd-scrub-repair/2/new.json 2024-03-30T11:30:55.984 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:30:55.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:30:55.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDv9wdmKIYOJxAAJ3i8DBQzT8Ul9PkelBZuHg== --osd-uuid dcdeba07-682b-453c-868b-832e3c6c50fd 2024-03-30T11:30:56.024 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.030+0000 7f22e720d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:56.069 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.075+0000 7f22e720d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:56.069 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.076+0000 7f22e720d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:30:56.070 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.076+0000 7f22e720d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:30:56.070 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.076+0000 7f22e720d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:30:56.070 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.076+0000 7f22e720d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:30:56.131 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:30:56.137+0000 7f22e720d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:31:02.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:31:02.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:31:02.194 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:31:02.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:31:02.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:31:02.561 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:31:02.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:31:02.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:31:02.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:31:02.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:31:02.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:31:02.587 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:02.594+0000 7f83ff613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:02.588 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:02.596+0000 7f83ff613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:02.590 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:02.597+0000 7f83ff613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:02.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:31:02.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:31:02.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:31:02.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:31:02.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:31:02.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:02.842 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:31:02.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:31:02.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:02.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:03.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:03.122 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:03.129+0000 7f83ff613640 -1 Falling back to public interface 2024-03-30T11:31:04.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:04.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:04.123 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:31:04.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:31:04.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:04.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:04.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:05.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:05.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:05.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:31:05.419 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:31:05.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:05.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:05.467 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:05.474+0000 7f83ff613640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:31:05.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:06.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:06.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:06.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:31:06.768 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:31:06.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:06.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:07.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:08.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:08.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:08.161 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:31:08.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:31:08.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:08.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:08.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:09.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:09.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:09.523 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:31:09.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:31:09.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:09.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:09.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:10.819 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:31:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:31:10.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:10.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:11.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:12.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:12.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:12.143 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:31:12.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:31:12.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:12.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:12.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:13.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:13.536 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:31:13.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:13.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:31:13.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:13.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:13.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:14.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:14.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:14.830 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:31:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:31:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:15.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:16.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:16.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:16.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:31:16.195 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:31:16.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:16.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:16.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:17.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:17.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:17.498 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:31:17.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:31:17.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:17.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:17.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:18.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:18.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:18.935 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:31:18.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:31:18.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:18.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:19.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:20.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:20.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:20.446 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:31:20.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:31:20.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:20.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:21.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:22.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:22.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:22.287 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:31:22.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:31:22.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:22.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:22.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:23.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:23.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:23.714 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:31:23.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:31:23.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:23.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:24.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:25.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:25.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:25.161 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:31:25.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:31:25.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:25.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:25.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:26.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:26.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:26.878 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:31:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:31:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:27.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:28.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:28.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:28.531 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:31:28.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:31:28.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:28.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:31:28.867 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3576770230,v1:127.0.0.1:6819/3576770230] [v2:127.0.0.1:6820/3576770230,v1:127.0.0.1:6821/3576770230] exists,up dcdeba07-682b-453c-868b-832e3c6c50fd 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-30T11:31:28.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:31:28.873 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:31:28.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:31:28.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:31:28.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:31:28.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:31:28.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:31:28.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 05caf833-bdd9-4407-a7b3-30c0a63bacdb' 2024-03-30T11:31:28.875 INFO:tasks.workunit.client.0.smithi003.stdout:add osd3 05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:31:28.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:31:28.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQ+AdmuNxyNRAAGyAcWb4v9alqPBNLAWL4DQ== 2024-03-30T11:31:28.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQ+AdmuNxyNRAAGyAcWb4v9alqPBNLAWL4DQ=="}' 2024-03-30T11:31:28.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05caf833-bdd9-4407-a7b3-30c0a63bacdb -i td/osd-scrub-repair/3/new.json 2024-03-30T11:31:29.273 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:31:29.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-30T11:31:29.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQ+AdmuNxyNRAAGyAcWb4v9alqPBNLAWL4DQ== --osd-uuid 05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:31:29.313 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.320+0000 7f38e4613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:29.357 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.364+0000 7f38e4613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:29.358 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.365+0000 7f38e4613640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:29.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.366+0000 7f38e4613640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:31:29.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.366+0000 7f38e4613640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:31:29.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.366+0000 7f38e4613640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:31:29.432 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:29.439+0000 7f38e4613640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-30T11:31:35.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-30T11:31:35.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:31:35.347 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd3 key to auth repository 2024-03-30T11:31:35.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-30T11:31:35.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:31:35.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-30T11:31:35.716 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.3 2024-03-30T11:31:35.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:31:35.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:31:35.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:31:35.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:31:35.742 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:35.749+0000 7fc2ed219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:35.743 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:35.751+0000 7fc2ed219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:35.744 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:35.752+0000 7fc2ed219640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:31:36.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-30T11:31:36.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:31:36.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:31:36.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:31:36.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:31:36.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:36.002 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:31:36.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:31:36.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:36.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:36.258 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:36.265+0000 7fc2ed219640 -1 Falling back to public interface 2024-03-30T11:31:36.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:37.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:31:37.302 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:31:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:37.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:38.599 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:31:38.606+0000 7fc2ed219640 -1 osd.3 0 log_to_monitors true 2024-03-30T11:31:38.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:38.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:38.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:31:38.600 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:31:38.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:38.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:38.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:39.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:39.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:39.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:31:39.928 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:31:39.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:39.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:40.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:41.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:41.254 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:31:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:31:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:41.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:41.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:42.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:42.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:42.591 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:31:42.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:31:42.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:42.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:42.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:43.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:43.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:43.885 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:31:43.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:31:43.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:43.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:44.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:45.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:45.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:45.223 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:31:45.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:31:45.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:45.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:45.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:46.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:46.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:46.528 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:31:46.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:31:46.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:46.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:46.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:47.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:47.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:47.822 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:31:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:31:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:48.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:49.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:49.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:49.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:31:49.121 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:31:49.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:49.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:49.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:50.536 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:31:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:31:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:50.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:50.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:51.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:51.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:51.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:31:51.829 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:31:51.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:51.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:52.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:53.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:53.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:53.391 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:31:53.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:31:53.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:53.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:53.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:54.971 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:31:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:31:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:55.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:56.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:56.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:56.702 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:31:56.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:31:56.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:56.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:56.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:57.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:57.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:57.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:31:57.996 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:31:57.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:57.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:31:58.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:31:59.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:31:59.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:31:59.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:31:59.619 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:31:59.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:31:59.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:32:00.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:01.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:01.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:01.063 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:32:01.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:32:01.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:01.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:32:01.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:02.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:02.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:02.398 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:32:02.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:32:02.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:02.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:32:02.690 INFO:tasks.workunit.client.0.smithi003.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2029604645,v1:127.0.0.1:6827/2029604645] [v2:127.0.0.1:6828/2029604645,v1:127.0.0.1:6829/2029604645] exists,up 05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:32:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:32:02.695 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:32:02.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd1fb0e9-e92c-420d-85c8-5b3e6e2275b5 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 dd1fb0e9-e92c-420d-85c8-5b3e6e2275b5' 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stdout:add osd4 dd1fb0e9-e92c-420d-85c8-5b3e6e2275b5 2024-03-30T11:32:02.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:32:02.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAy+AdmbO7PKhAAYIuCP03fKFAWGhkl1DO7Aw== 2024-03-30T11:32:02.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAy+AdmbO7PKhAAYIuCP03fKFAWGhkl1DO7Aw=="}' 2024-03-30T11:32:02.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd1fb0e9-e92c-420d-85c8-5b3e6e2275b5 -i td/osd-scrub-repair/4/new.json 2024-03-30T11:32:03.041 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:32:03.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-30T11:32:03.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAy+AdmbO7PKhAAYIuCP03fKFAWGhkl1DO7Aw== --osd-uuid dd1fb0e9-e92c-420d-85c8-5b3e6e2275b5 2024-03-30T11:32:03.081 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.088+0000 7fcc2820b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:03.125 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.132+0000 7fcc2820b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:03.126 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.133+0000 7fcc2820b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:03.127 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.134+0000 7fcc2820b640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-30T11:32:03.127 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.134+0000 7fcc2820b640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-30T11:32:03.127 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.134+0000 7fcc2820b640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-30T11:32:03.217 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:03.224+0000 7fcc2820b640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-30T11:32:08.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-30T11:32:08.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:32:08.977 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd4 key to auth repository 2024-03-30T11:32:08.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-30T11:32:08.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:32:09.348 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.4 2024-03-30T11:32:09.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-30T11:32:09.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:32:09.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:32:09.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:32:09.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:32:09.374 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:09.380+0000 7f5e40e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:09.376 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:09.382+0000 7f5e40e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:09.377 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:09.383+0000 7f5e40e13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-30T11:32:09.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:32:09.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-30T11:32:09.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:32:09.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:32:09.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:09.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:32:09.633 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:32:09.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:09.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:09.906 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:09.912+0000 7f5e40e13640 -1 Falling back to public interface 2024-03-30T11:32:09.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:10.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:10.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:10.913 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:32:10.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:32:10.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:10.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:11.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:12.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:12.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:12.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:32:12.219 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:32:12.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:12.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:12.341 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:12.347+0000 7f5e40e13640 -1 osd.4 0 log_to_monitors true 2024-03-30T11:32:12.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:13.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:13.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:13.539 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:32:13.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:32:13.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:13.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:13.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:14.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:14.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:32:14.831 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:32:14.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:14.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:15.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:16.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:16.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:16.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:32:16.142 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:32:16.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:16.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:16.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:17.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:17.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:17.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:32:17.435 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:32:17.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:17.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:17.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:18.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:18.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:18.729 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:32:18.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:32:18.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:18.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:19.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:20.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:20.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:20.022 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:32:20.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:32:20.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:20.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:20.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:21.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:21.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:21.318 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:32:21.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:32:21.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:21.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:22.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:22.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:22.636 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:32:22.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:32:22.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:22.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:22.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:23.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:23.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:23.963 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:32:23.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:32:23.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:23.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:24.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:25.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:25.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:25.670 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:32:25.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:32:25.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:25.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:26.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:27.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:27.207 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:32:27.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:27.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:32:27.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:27.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:27.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:28.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:28.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:28.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:32:28.851 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:32:28.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:28.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:29.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:30.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:30.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:30.605 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:32:30.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:32:30.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:30.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:31.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:32.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:32.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:32.405 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:32:32.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:32:32.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:32.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:33.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:34.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:34.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:34.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:32:34.069 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:32:34.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:34.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:35.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:35.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:35.362 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:32:35.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:32:35.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:35.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:32:35.666 INFO:tasks.workunit.client.0.smithi003.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1795858860,v1:127.0.0.1:6835/1795858860] [v2:127.0.0.1:6836/1795858860,v1:127.0.0.1:6837/1795858860] exists,up dd1fb0e9-e92c-420d-85c8-5b3e6e2275b5 2024-03-30T11:32:35.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:32:35.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:32:35.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:32:35.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:32:35.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:32:35.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:32:35.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:32:35.673 INFO:tasks.workunit.client.0.smithi003.stdout:add osd5 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:32:35.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf' 2024-03-30T11:32:35.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:32:35.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBT+AdmVABqKRAAKxBlK5YH1kwYvOVmbhjiwA== 2024-03-30T11:32:35.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBT+AdmVABqKRAAKxBlK5YH1kwYvOVmbhjiwA=="}' 2024-03-30T11:32:35.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf -i td/osd-scrub-repair/5/new.json 2024-03-30T11:32:36.022 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:32:36.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-30T11:32:36.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBT+AdmVABqKRAAKxBlK5YH1kwYvOVmbhjiwA== --osd-uuid 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:32:36.061 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.068+0000 7f26ca415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:36.106 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.113+0000 7f26ca415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:36.107 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.114+0000 7f26ca415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:36.108 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.115+0000 7f26ca415640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-30T11:32:36.108 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.115+0000 7f26ca415640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-30T11:32:36.108 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.115+0000 7f26ca415640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-30T11:32:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:36.188+0000 7f26ca415640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-30T11:32:42.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-30T11:32:42.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:32:42.380 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd5 key to auth repository 2024-03-30T11:32:42.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-30T11:32:42.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:32:42.750 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.5 2024-03-30T11:32:42.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-30T11:32:42.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:32:42.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:32:42.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:32:42.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:32:42.776 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:42.783+0000 7fa313811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:42.778 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:42.785+0000 7fa313811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:42.779 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:42.787+0000 7fa313811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:32:43.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-30T11:32:43.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:32:43.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:32:43.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:32:43.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:32:43.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:43.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:32:43.052 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:32:43.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:43.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:43.308+0000 7fa313811640 -1 Falling back to public interface 2024-03-30T11:32:43.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:44.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:44.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:44.333 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:32:44.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:32:44.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:44.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:44.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:45.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:45.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:45.626 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:32:45.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:32:45.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:45.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:45.649 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:32:45.656+0000 7fa313811640 -1 osd.5 0 log_to_monitors true 2024-03-30T11:32:45.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:46.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:46.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:46.959 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:32:46.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:32:46.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:46.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:47.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:48.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:48.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:48.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:32:48.264 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:32:48.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:48.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:48.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:49.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:49.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:49.660 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:32:49.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:32:49.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:49.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:49.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:50.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:50.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:50.953 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:32:50.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:32:50.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:50.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:51.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:52.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:52.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:52.245 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:32:52.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:32:52.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:52.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:52.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:53.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:53.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:53.679 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:32:53.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:32:53.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:53.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:53.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:54.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:54.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:54.974 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:32:54.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:32:54.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:54.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:55.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:56.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:56.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:56.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:32:56.270 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:32:56.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:56.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:56.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:57.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:57.700 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:32:57.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:57.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:32:57.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:57.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:58.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:32:59.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:32:59.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:32:59.019 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:32:59.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:32:59.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:32:59.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:32:59.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:00.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:00.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:00.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:33:00.431 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:33:00.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:00.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:33:00.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:01.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:01.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:01.983 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:33:01.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:33:01.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:01.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:33:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:03.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:03.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:03.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:33:03.553 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:33:03.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:03.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:33:04.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:05.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:05.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:05.133 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:33:05.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:33:05.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:05.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:33:05.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:06.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:06.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:06.923 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:33:06.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:33:06.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:06.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:33:07.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:08.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:08.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:08.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:33:08.526 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:33:08.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:08.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:33:08.861 INFO:tasks.workunit.client.0.smithi003.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1227714145,v1:127.0.0.1:6843/1227714145] [v2:127.0.0.1:6844/1227714145,v1:127.0.0.1:6845/1227714145] exists,up 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-30T11:33:08.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:33:08.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:33:08.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=90f1ddcc-8d80-4e2d-951f-c4b28982e51d 2024-03-30T11:33:08.869 INFO:tasks.workunit.client.0.smithi003.stdout:add osd6 90f1ddcc-8d80-4e2d-951f-c4b28982e51d 2024-03-30T11:33:08.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 90f1ddcc-8d80-4e2d-951f-c4b28982e51d' 2024-03-30T11:33:08.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:33:08.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0+AdmE+EMNRAAzKxumhGT6GxZYZjox8o20w== 2024-03-30T11:33:08.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0+AdmE+EMNRAAzKxumhGT6GxZYZjox8o20w=="}' 2024-03-30T11:33:08.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 90f1ddcc-8d80-4e2d-951f-c4b28982e51d -i td/osd-scrub-repair/6/new.json 2024-03-30T11:33:09.215 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:33:09.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-30T11:33:09.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0+AdmE+EMNRAAzKxumhGT6GxZYZjox8o20w== --osd-uuid 90f1ddcc-8d80-4e2d-951f-c4b28982e51d 2024-03-30T11:33:09.254 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.261+0000 7ff662608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:09.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.273+0000 7ff662608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:09.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.274+0000 7ff662608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.274+0000 7ff662608640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-30T11:33:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.274+0000 7ff662608640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-30T11:33:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.274+0000 7ff662608640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-30T11:33:09.379 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:09.386+0000 7ff662608640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-30T11:33:15.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-30T11:33:15.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:33:15.391 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd6 key to auth repository 2024-03-30T11:33:15.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-30T11:33:15.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:33:15.758 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.6 2024-03-30T11:33:15.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-30T11:33:15.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:33:15.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:33:15.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:33:15.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:33:15.784 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:15.791+0000 7f1fed60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:15.793+0000 7f1fed60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:15.787 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:15.794+0000 7f1fed60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:33:16.041 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:33:16.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:16.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:16.319+0000 7f1fed60d640 -1 Falling back to public interface 2024-03-30T11:33:16.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:17.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:17.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:17.322 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:33:17.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:33:17.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:17.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:17.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:18.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:18.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:18.616 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:33:18.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:33:18.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:18.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:18.644 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:18.650+0000 7f1fed60d640 -1 osd.6 0 log_to_monitors true 2024-03-30T11:33:18.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:33:19.961 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:33:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:20.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:21.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:21.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:21.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:33:21.321 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:33:21.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:21.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:21.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:22.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:22.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:22.709 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:33:22.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:33:22.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:22.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:23.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:24.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:24.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:33:24.019 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:33:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:24.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:25.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:25.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:25.439 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:33:25.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:33:25.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:25.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:26.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:26.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:26.741 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:33:26.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:33:26.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:26.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:27.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:28.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:28.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:28.033 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:33:28.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:33:28.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:28.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:28.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:29.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:29.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:29.329 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:33:29.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:33:29.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:29.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:29.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:30.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:30.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:30.675 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:33:30.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:33:30.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:30.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:31.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:32.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:32.152 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:33:32.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:32.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:33:32.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:32.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:33.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:33.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:33.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:33:33.578 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:33:33.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:33.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:34.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:35.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:35.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:35.155 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:33:35.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:33:35.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:35.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:35.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:36.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:36.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:36.560 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:33:36.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:33:36.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:36.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:37.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:38.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:38.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:38.380 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:33:38.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:33:38.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:38.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:38.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:39.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:39.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:39.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:33:39.948 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:33:39.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:39.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:40.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:33:41.497 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:33:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:41.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:33:41.800 INFO:tasks.workunit.client.0.smithi003.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1604475039,v1:127.0.0.1:6851/1604475039] [v2:127.0.0.1:6852/1604475039,v1:127.0.0.1:6853/1604475039] exists,up 90f1ddcc-8d80-4e2d-951f-c4b28982e51d 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:33:41.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:33:41.805 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:33:41.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:33:41.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:33:41.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-30T11:33:41.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:33:41.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c52e9aea-00cd-4fc5-bd56-200a3f02dc21 2024-03-30T11:33:41.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 c52e9aea-00cd-4fc5-bd56-200a3f02dc21' 2024-03-30T11:33:41.807 INFO:tasks.workunit.client.0.smithi003.stdout:add osd7 c52e9aea-00cd-4fc5-bd56-200a3f02dc21 2024-03-30T11:33:41.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:33:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCV+Adme+ZjMRAAC/nnpcN3Rlk/QEkKAVYjKQ== 2024-03-30T11:33:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCV+Adme+ZjMRAAC/nnpcN3Rlk/QEkKAVYjKQ=="}' 2024-03-30T11:33:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c52e9aea-00cd-4fc5-bd56-200a3f02dc21 -i td/osd-scrub-repair/7/new.json 2024-03-30T11:33:42.193 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:33:42.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-30T11:33:42.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCV+Adme+ZjMRAAC/nnpcN3Rlk/QEkKAVYjKQ== --osd-uuid c52e9aea-00cd-4fc5-bd56-200a3f02dc21 2024-03-30T11:33:42.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.240+0000 7fefc060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:42.290 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.297+0000 7fefc060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:42.291 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.298+0000 7fefc060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:42.291 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.298+0000 7fefc060d640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-30T11:33:42.291 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.298+0000 7fefc060d640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-30T11:33:42.291 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.298+0000 7fefc060d640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-30T11:33:42.373 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:42.380+0000 7fefc060d640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-30T11:33:48.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-30T11:33:48.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:33:48.863 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd7 key to auth repository 2024-03-30T11:33:48.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-30T11:33:48.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:33:49.236 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.7 2024-03-30T11:33:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-30T11:33:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:33:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:33:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:33:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:33:49.262 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:49.269+0000 7f7545801640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:49.264 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:49.271+0000 7f7545801640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:49.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:49.272+0000 7f7545801640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:33:49.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:33:49.561 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:33:49.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:49.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:49.792+0000 7f7545801640 -1 Falling back to public interface 2024-03-30T11:33:49.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:50.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:50.805 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:33:50.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:33:50.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:50.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:52.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:52.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:52.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:33:52.098 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:33:52.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:52.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:52.190 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:33:52.198+0000 7f7545801640 -1 osd.7 0 log_to_monitors true 2024-03-30T11:33:52.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:53.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:53.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:53.394 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:33:53.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:33:53.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:53.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:53.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:54.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:54.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:54.691 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:33:54.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:33:54.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:54.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:54.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:55.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:55.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:55.984 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:33:55.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:33:55.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:55.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:56.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:57.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:57.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:57.301 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:33:57.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:33:57.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:57.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:57.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:58.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:58.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:58.646 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:33:58.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:33:58.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:58.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:33:58.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:33:59.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:33:59.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:33:59.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:33:59.941 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:33:59.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:33:59.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:00.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:01.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:01.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:01.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:34:01.276 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:34:01.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:01.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:01.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:02.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:02.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:02.660 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:34:02.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:34:02.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:02.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:02.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:03.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:03.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:03.962 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:34:03.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:34:03.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:03.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:04.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:05.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:05.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:05.255 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:34:05.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:34:05.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:05.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:05.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:06.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:06.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:06.964 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:34:06.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:34:06.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:06.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:07.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:08.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:08.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:08.268 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:34:08.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:34:08.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:08.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:08.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:09.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:09.961 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:34:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:34:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:09.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:10.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:11.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:11.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:11.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:34:11.448 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:34:11.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:11.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:34:13.175 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:34:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:15.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:15.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:15.043 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:34:15.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:34:15.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:15.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:15.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:16.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:16.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:16.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:34:16.359 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:34:16.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:16.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:34:16.650 INFO:tasks.workunit.client.0.smithi003.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2515969321,v1:127.0.0.1:6859/2515969321] [v2:127.0.0.1:6860/2515969321,v1:127.0.0.1:6861/2515969321] exists,up c52e9aea-00cd-4fc5-bd56-200a3f02dc21 2024-03-30T11:34:16.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:34:16.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:34:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=834d9857-0501-4687-a419-1b2bd5934b4b 2024-03-30T11:34:16.657 INFO:tasks.workunit.client.0.smithi003.stdout:add osd8 834d9857-0501-4687-a419-1b2bd5934b4b 2024-03-30T11:34:16.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 834d9857-0501-4687-a419-1b2bd5934b4b' 2024-03-30T11:34:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:34:16.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4+AdmeolWKBAAfVeCfGEZz9noaq2nW6zOew== 2024-03-30T11:34:16.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4+AdmeolWKBAAfVeCfGEZz9noaq2nW6zOew=="}' 2024-03-30T11:34:16.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 834d9857-0501-4687-a419-1b2bd5934b4b -i td/osd-scrub-repair/8/new.json 2024-03-30T11:34:17.109 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:34:17.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-30T11:34:17.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4+AdmeolWKBAAfVeCfGEZz9noaq2nW6zOew== --osd-uuid 834d9857-0501-4687-a419-1b2bd5934b4b 2024-03-30T11:34:17.148 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.155+0000 7f4fca424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:17.159 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.167+0000 7f4fca424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:17.160 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.168+0000 7f4fca424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:17.161 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.168+0000 7f4fca424640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-30T11:34:17.161 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.168+0000 7f4fca424640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-30T11:34:17.161 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.168+0000 7f4fca424640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-30T11:34:17.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:17.275+0000 7f4fca424640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-30T11:34:23.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-30T11:34:23.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:34:23.453 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd8 key to auth repository 2024-03-30T11:34:23.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-30T11:34:23.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:34:23.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-30T11:34:23.823 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.8 2024-03-30T11:34:23.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:34:23.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:34:23.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:34:23.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:34:23.851 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:23.858+0000 7fade4c2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:23.852 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:23.860+0000 7fade4c2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:23.853 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:23.861+0000 7fade4c2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:24.105 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:34:24.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:34:24.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:24.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:24.373 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:24.380+0000 7fade4c2a640 -1 Falling back to public interface 2024-03-30T11:34:24.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:25.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:25.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:25.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:34:25.395 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:34:25.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:25.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:25.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:26.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:26.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:26.686 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:34:26.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:34:26.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:26.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:26.716 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:26.723+0000 7fade4c2a640 -1 osd.8 0 log_to_monitors true 2024-03-30T11:34:27.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:28.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:28.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:28.014 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:34:28.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:34:28.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:28.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:28.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:29.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:29.388 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:34:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:34:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:29.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:30.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:30.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:30.680 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:34:30.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:34:30.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:30.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:31.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:32.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:32.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:32.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:34:32.026 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:34:32.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:32.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:32.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:33.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:33.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:33.318 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:34:33.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:34:33.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:33.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:33.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:34.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:34.615 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:34:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:34:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:34.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:34.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:35.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:35.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:35.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:34:35.908 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:34:35.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:35.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:36.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:37.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:37.202 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:34:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:34:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:37.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:38.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:38.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:38.498 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:34:38.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:34:38.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:38.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:38.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:39.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:39.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:39.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:34:39.790 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:34:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:40.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:41.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:41.426 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:34:41.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:34:41.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:41.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:41.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:42.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:42.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:42.827 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:34:42.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:34:42.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:42.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:43.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:44.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:44.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:44.646 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:34:44.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:34:44.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:44.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:45.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:46.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:46.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:46.376 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:34:46.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:34:46.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:46.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:47.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:48.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:48.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:48.036 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:34:48.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:34:48.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:48.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:48.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:49.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:49.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:49.667 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:34:49.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:34:49.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:49.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:50.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:34:51.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:34:51.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:51.084 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:34:51.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:34:51.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:51.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:34:51.376 INFO:tasks.workunit.client.0.smithi003.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/367669373,v1:127.0.0.1:6867/367669373] [v2:127.0.0.1:6868/367669373,v1:127.0.0.1:6869/367669373] exists,up 834d9857-0501-4687-a419-1b2bd5934b4b 2024-03-30T11:34:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:34:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:34:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:34:51.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:34:51.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:34:51.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:34:51.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 0cddbc61-5758-4439-8e42-fa1b67b5ceae' 2024-03-30T11:34:51.384 INFO:tasks.workunit.client.0.smithi003.stdout:add osd9 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:34:51.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:34:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDb+Adm9GsaGBAAce2ESzDep194TtMJ44TlLA== 2024-03-30T11:34:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDb+Adm9GsaGBAAce2ESzDep194TtMJ44TlLA=="}' 2024-03-30T11:34:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cddbc61-5758-4439-8e42-fa1b67b5ceae -i td/osd-scrub-repair/9/new.json 2024-03-30T11:34:51.858 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:34:51.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-30T11:34:51.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDb+Adm9GsaGBAAce2ESzDep194TtMJ44TlLA== --osd-uuid 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:34:51.898 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:51.905+0000 7fe8c020d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:51.943 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:51.950+0000 7fe8c020d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:51.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:51.951+0000 7fe8c020d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:51.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:51.951+0000 7fe8c020d640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-30T11:34:51.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:51.951+0000 7fe8c020d640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-30T11:34:51.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:51.951+0000 7fe8c020d640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-30T11:34:52.035 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:52.042+0000 7fe8c020d640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-30T11:34:58.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-30T11:34:58.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:34:58.076 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd9 key to auth repository 2024-03-30T11:34:58.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-30T11:34:58.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:34:58.450 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.9 2024-03-30T11:34:58.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-30T11:34:58.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:34:58.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:34:58.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:34:58.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:34:58.476 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:58.483+0000 7fa6f3415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:58.478 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:58.485+0000 7fa6f3415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:58.479 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:58.486+0000 7fa6f3415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:34:58.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:34:58.732 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:34:58.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:34:58.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:34:59.011 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:34:59.018+0000 7fa6f3415640 -1 Falling back to public interface 2024-03-30T11:34:59.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:00.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:00.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:00.021 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:35:00.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:35:00.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:00.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:00.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:01.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:01.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:01.318 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:35:01.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:35:01.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:01.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:01.427 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:35:01.433+0000 7fa6f3415640 -1 osd.9 0 log_to_monitors true 2024-03-30T11:35:01.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:02.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:02.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:02.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:35:02.634 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:35:02.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:02.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:02.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:35:03.931 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:35:03.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:03.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:04.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:05.374 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:35:05.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:05.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:05.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:35:05.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:05.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:05.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:06.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:06.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:06.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:35:06.741 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:35:06.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:06.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:07.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:08.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:08.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:08.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:35:08.035 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:35:08.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:08.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:09.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:09.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:09.343 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:35:09.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:35:09.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:09.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:09.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:10.637 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:35:10.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:35:10.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:10.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:10.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:11.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:11.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:11.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:35:11.932 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:35:11.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:11.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:12.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:13.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:13.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:13.374 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:35:13.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:35:13.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:13.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:13.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:14.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:14.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:14.669 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:35:14.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:35:14.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:14.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:15.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:16.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:16.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:16.483 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:35:16.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:35:16.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:16.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:17.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:18.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:18.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:18.084 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:35:18.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:35:18.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:18.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:19.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:20.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:20.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:20.033 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:35:20.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:35:20.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:20.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:20.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:21.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:21.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:21.402 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:35:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:35:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:21.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:22.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:22.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:22.981 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:35:22.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:35:22.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:22.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:23.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:24.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:24.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:24.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:35:24.602 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:35:24.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:24.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:35:24.957 INFO:tasks.workunit.client.0.smithi003.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2184054147,v1:127.0.0.1:6875/2184054147] [v2:127.0.0.1:6876/2184054147,v1:127.0.0.1:6877/2184054147] exists,up 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:35:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:35:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:35:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:35:24.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-30T11:35:24.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:35:25.283 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:35:25.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:35:25.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:35:25.683 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:35:25.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:35:26.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:35:29.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-30T11:35:29.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:35:29.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:35:29.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:35:29.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:35:29.652 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:35:29.652 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:35:29.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:35:29.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:35:29.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:35:29.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:35:29.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:35:29.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:35:29.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:35:29.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:35:29.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:35:30.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:35:30.035 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:35:30.035 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:35:30.035 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:35:30.035 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:35:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-03-30T11:35:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-03-30T11:35:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-03-30T11:35:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.140 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:35:30.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-03-30T11:35:30.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-03-30T11:35:30.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022' 2024-03-30T11:35:30.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:35:30.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509490 2024-03-30T11:35:30.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509490 2024-03-30T11:35:30.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490' 2024-03-30T11:35:30.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:35:30.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345969 2024-03-30T11:35:30.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345969 2024-03-30T11:35:30.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969' 2024-03-30T11:35:30.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:35:30.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182439 2024-03-30T11:35:30.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182439 2024-03-30T11:35:30.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969 4-107374182439' 2024-03-30T11:35:30.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:35:30.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018915 2024-03-30T11:35:30.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018915 2024-03-30T11:35:30.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969 4-107374182439 5-128849018915' 2024-03-30T11:35:30.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:35:30.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855385 2024-03-30T11:35:30.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855385 2024-03-30T11:35:30.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969 4-107374182439 5-128849018915 6-150323855385' 2024-03-30T11:35:30.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:35:30.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-03-30T11:35:30.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-03-30T11:35:30.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969 4-107374182439 5-128849018915 6-150323855385 7-171798691861' 2024-03-30T11:35:30.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:30.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:35:31.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528332 2024-03-30T11:35:31.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528332 2024-03-30T11:35:31.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969 4-107374182439 5-128849018915 6-150323855385 7-171798691861 8-193273528332' 2024-03-30T11:35:31.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:31.014 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:35:31.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364807 2024-03-30T11:35:31.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364807 2024-03-30T11:35:31.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673022 2-64424509490 3-85899345969 4-107374182439 5-128849018915 6-150323855385 7-171798691861 8-193273528332 9-214748364807' 2024-03-30T11:35:31.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:31.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-03-30T11:35:31.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:31.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:35:31.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-03-30T11:35:31.121 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:31.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-03-30T11:35:31.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-03-30T11:35:31.122 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836547 2024-03-30T11:35:31.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:35:31.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836547 2024-03-30T11:35:31.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:35:32.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:35:32.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:35:32.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836547 2024-03-30T11:35:32.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:32.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-03-30T11:35:32.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:32.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:35:32.704 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-03-30T11:35:32.704 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:32.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-03-30T11:35:32.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-03-30T11:35:32.705 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949673022 2024-03-30T11:35:32.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:35:33.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673022 2024-03-30T11:35:33.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:33.002 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509490 2024-03-30T11:35:33.002 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:33.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:35:33.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509490 2024-03-30T11:35:33.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509490 2024-03-30T11:35:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2024-03-30T11:35:33.005 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509490 2024-03-30T11:35:33.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:35:33.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509490 -lt 64424509490 2024-03-30T11:35:33.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:33.290 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345969 2024-03-30T11:35:33.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:35:33.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345969 2024-03-30T11:35:33.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:33.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345969 2024-03-30T11:35:33.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345969' 2024-03-30T11:35:33.294 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345969 2024-03-30T11:35:33.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:35:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345969 -lt 85899345969 2024-03-30T11:35:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:33.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182439 2024-03-30T11:35:33.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:33.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:35:33.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182439 2024-03-30T11:35:33.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:33.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182439 2024-03-30T11:35:33.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2024-03-30T11:35:33.598 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.4 seq 107374182439 2024-03-30T11:35:33.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:35:33.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182439 -lt 107374182439 2024-03-30T11:35:33.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:33.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018915 2024-03-30T11:35:33.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:33.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:35:33.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018915 2024-03-30T11:35:33.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:33.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018915 2024-03-30T11:35:33.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-03-30T11:35:33.883 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.5 seq 128849018915 2024-03-30T11:35:33.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:35:34.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018915 -lt 128849018915 2024-03-30T11:35:34.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:34.193 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855385 2024-03-30T11:35:34.193 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:34.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:35:34.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855385 2024-03-30T11:35:34.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:34.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855385 2024-03-30T11:35:34.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855385' 2024-03-30T11:35:34.196 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.6 seq 150323855385 2024-03-30T11:35:34.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:35:34.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855386 -lt 150323855385 2024-03-30T11:35:34.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:34.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-03-30T11:35:34.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:34.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:35:34.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-03-30T11:35:34.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:34.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-03-30T11:35:34.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-03-30T11:35:34.650 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.7 seq 171798691861 2024-03-30T11:35:34.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:35:34.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691861 2024-03-30T11:35:34.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:34.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528332 2024-03-30T11:35:34.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:34.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:35:34.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528332 2024-03-30T11:35:34.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:34.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528332 2024-03-30T11:35:34.987 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.8 seq 193273528332 2024-03-30T11:35:34.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528332' 2024-03-30T11:35:34.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:35:35.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528333 -lt 193273528332 2024-03-30T11:35:35.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:35.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364807 2024-03-30T11:35:35.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:35.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:35:35.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364807 2024-03-30T11:35:35.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:35.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364807 2024-03-30T11:35:35.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364807' 2024-03-30T11:35:35.277 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.9 seq 214748364807 2024-03-30T11:35:35.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:35:35.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364807 -lt 214748364807 2024-03-30T11:35:35.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:35:35.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:35:35.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:35:35.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:35:36.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:35:36.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:35:36.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:35:36.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:35:36.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-30T11:35:36.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:35:36.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:35:37.666 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:35:37.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:35:38.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-30T11:35:38.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:35:38.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:35:38.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:35:38.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:35:38.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:35:38.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:35:38.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:35:38.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:35:38.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:35:38.777 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:35:39.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:35:39.082 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:35:39.082 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:35:39.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-03-30T11:35:39.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-03-30T11:35:39.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-03-30T11:35:39.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.188 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:35:39.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-03-30T11:35:39.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-03-30T11:35:39.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025' 2024-03-30T11:35:39.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:35:39.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509493 2024-03-30T11:35:39.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509493 2024-03-30T11:35:39.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493' 2024-03-30T11:35:39.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:35:39.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345972 2024-03-30T11:35:39.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345972 2024-03-30T11:35:39.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972' 2024-03-30T11:35:39.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:35:39.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182441 2024-03-30T11:35:39.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182441 2024-03-30T11:35:39.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972 4-107374182441' 2024-03-30T11:35:39.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:35:39.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-03-30T11:35:39.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-03-30T11:35:39.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972 4-107374182441 5-128849018918' 2024-03-30T11:35:39.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:35:39.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855388 2024-03-30T11:35:39.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855388 2024-03-30T11:35:39.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972 4-107374182441 5-128849018918 6-150323855388' 2024-03-30T11:35:39.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:35:39.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691864 2024-03-30T11:35:39.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691864 2024-03-30T11:35:39.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972 4-107374182441 5-128849018918 6-150323855388 7-171798691864' 2024-03-30T11:35:39.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:39.956 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:35:40.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528335 2024-03-30T11:35:40.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528335 2024-03-30T11:35:40.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972 4-107374182441 5-128849018918 6-150323855388 7-171798691864 8-193273528335' 2024-03-30T11:35:40.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:40.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:35:40.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364810 2024-03-30T11:35:40.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364810 2024-03-30T11:35:40.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673025 2-64424509493 3-85899345972 4-107374182441 5-128849018918 6-150323855388 7-171798691864 8-193273528335 9-214748364810' 2024-03-30T11:35:40.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:40.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-03-30T11:35:40.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:40.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:35:40.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-03-30T11:35:40.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:40.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-03-30T11:35:40.169 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836550 2024-03-30T11:35:40.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-03-30T11:35:40.170 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:35:40.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-03-30T11:35:40.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:40.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-03-30T11:35:40.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:40.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:35:40.498 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-03-30T11:35:40.499 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:40.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-03-30T11:35:40.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-03-30T11:35:40.500 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949673025 2024-03-30T11:35:40.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:35:40.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-03-30T11:35:40.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:40.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509493 2024-03-30T11:35:40.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:40.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:35:40.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509493 2024-03-30T11:35:40.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:40.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509493 2024-03-30T11:35:40.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-03-30T11:35:40.821 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509493 2024-03-30T11:35:40.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:35:41.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509493 -lt 64424509493 2024-03-30T11:35:41.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:41.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345972 2024-03-30T11:35:41.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:41.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:35:41.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345972 2024-03-30T11:35:41.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:41.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345972 2024-03-30T11:35:41.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345972' 2024-03-30T11:35:41.155 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345972 2024-03-30T11:35:41.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:35:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345972 -lt 85899345972 2024-03-30T11:35:41.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:41.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182441 2024-03-30T11:35:41.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:41.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:35:41.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182441 2024-03-30T11:35:41.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:41.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182441 2024-03-30T11:35:41.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2024-03-30T11:35:41.484 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.4 seq 107374182441 2024-03-30T11:35:41.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:35:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182442 -lt 107374182441 2024-03-30T11:35:41.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:41.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018918 2024-03-30T11:35:41.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:41.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:35:41.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018918 2024-03-30T11:35:41.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-03-30T11:35:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-03-30T11:35:41.770 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.5 seq 128849018918 2024-03-30T11:35:41.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:35:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018918 -lt 128849018918 2024-03-30T11:35:42.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:42.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855388 2024-03-30T11:35:42.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:42.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:35:42.055 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855388 2024-03-30T11:35:42.055 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855388 2024-03-30T11:35:42.056 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.6 seq 150323855388 2024-03-30T11:35:42.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-03-30T11:35:42.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:35:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855388 2024-03-30T11:35:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:42.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691864 2024-03-30T11:35:42.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:35:42.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691864 2024-03-30T11:35:42.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:42.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691864 2024-03-30T11:35:42.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2024-03-30T11:35:42.352 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.7 seq 171798691864 2024-03-30T11:35:42.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:35:42.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691864 2024-03-30T11:35:42.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:42.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528335 2024-03-30T11:35:42.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:35:42.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528335 2024-03-30T11:35:42.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:42.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528335 2024-03-30T11:35:42.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-03-30T11:35:42.638 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.8 seq 193273528335 2024-03-30T11:35:42.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:35:42.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528335 2024-03-30T11:35:42.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:42.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364810 2024-03-30T11:35:42.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:42.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:35:42.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364810 2024-03-30T11:35:42.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:42.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364810 2024-03-30T11:35:42.924 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.9 seq 214748364810 2024-03-30T11:35:42.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-03-30T11:35:42.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:35:43.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364810 -lt 214748364810 2024-03-30T11:35:43.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:35:43.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:35:43.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:35:43.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:35:43.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:35:43.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:35:43.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:35:43.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:35:43.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:35:44.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:35:44.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:35:44.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:35:44.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:35:44.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:35:44.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:35:44.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:35:44.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:35:44.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:35:44.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:35:44.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:35:44.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:35:44.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:35:44.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:35:45.368 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:35:45.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:35:45.771 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:35:45.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:35:45.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:35:45.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:35:46.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-30T11:35:46.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:35:46.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:35:46.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:35:46.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:35:46.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-30T11:35:46.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-30T11:35:46.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T11:35:46.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-30T11:35:46.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-30T11:35:46.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:35:46.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T11:35:46.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:9 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:7' 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-30T11:35:46.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:35:46.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:35:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:35:47.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:35:47.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:35:47.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:35:47.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:35:47.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:35:47.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:35:47.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:35:47.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:35:49.263 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:35:49.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:35:49.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:35:49.784 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:35:49.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:35:49.785 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:35:49.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:35:49.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:35:49.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:35:49.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:35:49.812 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:35:49.818+0000 7f5167237640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:35:49.814 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:35:49.820+0000 7f5167237640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:35:49.815 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:35:49.821+0000 7f5167237640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:35:50.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:35:50.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:35:50.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:35:50.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:35:50.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:35:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:35:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:35:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:35:50.337 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:35:50.343+0000 7f5167237640 -1 Falling back to public interface 2024-03-30T11:35:50.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:51.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:35:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:35:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:51.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:35:51.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:52.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:52.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:52.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:35:52.692 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:35:52.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:52.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:35:52.720 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:35:52.727+0000 7f5167237640 -1 osd.3 67 log_to_monitors true 2024-03-30T11:35:53.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:35:54.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:35:54.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:35:54.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:35:54.025 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:35:54.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:35:54.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/1134283771,v1:127.0.0.1:6827/1134283771] [v2:127.0.0.1:6828/1134283771,v1:127.0.0.1:6829/1134283771] exists,up 05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:35:54.371 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:35:54.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:35:54.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:35:54.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:35:54.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:35:54.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:35:54.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:35:54.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:35:54.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:35:54.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:35:54.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:35:54.779 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:54.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:35:54.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836554 2024-03-30T11:35:54.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836554 2024-03-30T11:35:54.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554' 2024-03-30T11:35:54.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:54.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:35:54.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673029 2024-03-30T11:35:54.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673029 2024-03-30T11:35:54.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029' 2024-03-30T11:35:54.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:54.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:35:55.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509497 2024-03-30T11:35:55.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509497 2024-03-30T11:35:55.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497' 2024-03-30T11:35:55.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:35:55.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-03-30T11:35:55.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-03-30T11:35:55.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018' 2024-03-30T11:35:55.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:35:55.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182446 2024-03-30T11:35:55.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182446 2024-03-30T11:35:55.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018 4-107374182446' 2024-03-30T11:35:55.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:35:55.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018922 2024-03-30T11:35:55.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018922 2024-03-30T11:35:55.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018 4-107374182446 5-128849018922' 2024-03-30T11:35:55.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:35:55.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855392 2024-03-30T11:35:55.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855392 2024-03-30T11:35:55.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018 4-107374182446 5-128849018922 6-150323855392' 2024-03-30T11:35:55.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:35:55.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-03-30T11:35:55.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-03-30T11:35:55.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018 4-107374182446 5-128849018922 6-150323855392 7-171798691868' 2024-03-30T11:35:55.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:35:55.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528339 2024-03-30T11:35:55.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528339 2024-03-30T11:35:55.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018 4-107374182446 5-128849018922 6-150323855392 7-171798691868 8-193273528339' 2024-03-30T11:35:55.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:35:55.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:35:55.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364814 2024-03-30T11:35:55.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364814 2024-03-30T11:35:55.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509497 3-304942678018 4-107374182446 5-128849018922 6-150323855392 7-171798691868 8-193273528339 9-214748364814' 2024-03-30T11:35:55.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836554 2024-03-30T11:35:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:55.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:35:55.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836554 2024-03-30T11:35:55.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:55.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836554 2024-03-30T11:35:55.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-03-30T11:35:55.837 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836554 2024-03-30T11:35:55.837 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:35:56.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836554 2024-03-30T11:35:56.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:35:57.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:35:57.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:35:57.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836554 2024-03-30T11:35:57.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:57.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673029 2024-03-30T11:35:57.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:57.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:35:57.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673029 2024-03-30T11:35:57.420 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:57.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673029 2024-03-30T11:35:57.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-03-30T11:35:57.421 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673029 2024-03-30T11:35:57.421 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:35:57.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673030 -lt 42949673029 2024-03-30T11:35:57.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:57.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509497 2024-03-30T11:35:57.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:57.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:35:57.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509497 2024-03-30T11:35:57.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:57.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509497 2024-03-30T11:35:57.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-03-30T11:35:57.708 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509497 2024-03-30T11:35:57.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:35:57.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509497 2024-03-30T11:35:57.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:57.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-03-30T11:35:57.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:57.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:35:57.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-03-30T11:35:57.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:57.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-03-30T11:35:57.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-03-30T11:35:57.994 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678018 2024-03-30T11:35:57.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:35:58.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678019 -lt 304942678018 2024-03-30T11:35:58.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:58.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182446 2024-03-30T11:35:58.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:58.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:35:58.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182446 2024-03-30T11:35:58.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:58.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182446 2024-03-30T11:35:58.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-03-30T11:35:58.287 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182446 2024-03-30T11:35:58.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:35:58.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182446 -lt 107374182446 2024-03-30T11:35:58.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:58.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018922 2024-03-30T11:35:58.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:58.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:35:58.571 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018922 2024-03-30T11:35:58.571 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:58.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018922 2024-03-30T11:35:58.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-03-30T11:35:58.572 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 128849018922 2024-03-30T11:35:58.573 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:35:58.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018922 -lt 128849018922 2024-03-30T11:35:58.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:58.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855392 2024-03-30T11:35:58.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:58.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:35:58.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855392 2024-03-30T11:35:58.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:58.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855392 2024-03-30T11:35:58.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-03-30T11:35:58.858 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855392 2024-03-30T11:35:58.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:35:59.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855393 -lt 150323855392 2024-03-30T11:35:59.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:59.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-03-30T11:35:59.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:59.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:35:59.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-03-30T11:35:59.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:59.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-03-30T11:35:59.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-03-30T11:35:59.145 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691868 2024-03-30T11:35:59.145 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:35:59.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-03-30T11:35:59.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:59.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528339 2024-03-30T11:35:59.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:59.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:35:59.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528339 2024-03-30T11:35:59.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:59.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528339 2024-03-30T11:35:59.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-03-30T11:35:59.429 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528339 2024-03-30T11:35:59.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:35:59.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528339 2024-03-30T11:35:59.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:35:59.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364814 2024-03-30T11:35:59.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:35:59.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:35:59.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364814 2024-03-30T11:35:59.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:35:59.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364814 2024-03-30T11:35:59.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-03-30T11:35:59.713 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 214748364814 2024-03-30T11:35:59.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:35:59.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364814 -lt 214748364814 2024-03-30T11:35:59.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:35:59.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:35:59.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:00.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:36:00.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:36:00.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:36:00.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:36:00.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:36:00.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:36:00.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:36:00.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:36:00.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:36:00.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:36:00.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:00.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:00.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:36:00.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:36:00.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:36:00.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:36:00.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:36:00.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:36:00.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:36:00.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:01.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:01.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:35:37.523278+0000 2024-03-30T11:36:01.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:36:01.825 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:36:01.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:35:37.523278+0000 2024-03-30T11:36:01.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:36:01.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:35:37.523278+0000 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:01.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:02.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:35:37.523278+0000 '>' 2024-03-30T11:35:37.523278+0000 2024-03-30T11:36:02.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:36:03.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:36:03.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:03.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:03.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:03.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:03.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:03.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:35:37.523278+0000 '>' 2024-03-30T11:35:37.523278+0000 2024-03-30T11:36:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:36:04.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:36:04.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:04.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:04.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:04.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:04.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:04.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:04.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:02.588611+0000 '>' 2024-03-30T11:35:37.523278+0000 2024-03-30T11:36:04.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:36:04.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:36:04.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:36:04.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:36:04.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:36:04.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:36:05.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:36:05.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:36:05.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:36:05.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:36:05.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:36:05.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:36:05.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:36:05.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-30T11:36:07.080 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:36:07.080 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:36:07.080 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:36:07.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:36:07.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:36:07.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:36:07.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:36:07.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:36:07.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:36:07.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:36:07.351 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.9 2024-03-30T11:36:07.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:36:07.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:36:07.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:36:07.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:36:07.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:36:07.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:36:07.378 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:07.385+0000 7fca97c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:07.380 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:07.387+0000 7fca97c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:07.381 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:07.388+0000 7fca97c20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:36:07.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:07.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:07.898 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:07.905+0000 7fca97c20640 -1 Falling back to public interface 2024-03-30T11:36:07.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:08.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:08.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:08.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:36:08.929 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:36:08.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:08.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:09.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:10.104 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:10.111+0000 7fca97c20640 -1 osd.9 72 log_to_monitors true 2024-03-30T11:36:10.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:10.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:10.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:36:10.264 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:36:10.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:10.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/558723881,v1:127.0.0.1:6875/558723881] [v2:127.0.0.1:6876/558723881,v1:127.0.0.1:6877/558723881] exists,up 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:36:10.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:36:10.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:36:10.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:36:10.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:36:10.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:36:10.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:36:10.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:36:10.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:36:11.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:11.240 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:36:11.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-03-30T11:36:11.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-03-30T11:36:11.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-03-30T11:36:11.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:11.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:36:11.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-03-30T11:36:11.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-03-30T11:36:11.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034' 2024-03-30T11:36:11.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:11.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:36:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-03-30T11:36:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-03-30T11:36:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501' 2024-03-30T11:36:12.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:12.112 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:36:12.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678022 2024-03-30T11:36:12.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678022 2024-03-30T11:36:12.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022' 2024-03-30T11:36:12.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:12.362 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:36:12.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182450 2024-03-30T11:36:12.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182450 2024-03-30T11:36:12.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022 4-107374182450' 2024-03-30T11:36:12.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:12.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:36:12.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018926 2024-03-30T11:36:12.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018926 2024-03-30T11:36:12.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022 4-107374182450 5-128849018926' 2024-03-30T11:36:12.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:12.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:36:12.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-03-30T11:36:12.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-03-30T11:36:12.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022 4-107374182450 5-128849018926 6-150323855397' 2024-03-30T11:36:12.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:12.804 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:36:12.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-03-30T11:36:12.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-03-30T11:36:12.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022 4-107374182450 5-128849018926 6-150323855397 7-171798691873' 2024-03-30T11:36:12.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:12.909 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:36:13.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528344 2024-03-30T11:36:13.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528344 2024-03-30T11:36:13.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022 4-107374182450 5-128849018926 6-150323855397 7-171798691873 8-193273528344' 2024-03-30T11:36:13.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:13.028 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:36:13.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:36:13.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:36:13.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673034 2-64424509501 3-304942678022 4-107374182450 5-128849018926 6-150323855397 7-171798691873 8-193273528344 9-330712481794' 2024-03-30T11:36:13.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:13.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-03-30T11:36:13.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:13.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:36:13.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-03-30T11:36:13.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:13.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-03-30T11:36:13.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-03-30T11:36:13.158 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836558 2024-03-30T11:36:13.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:36:13.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836558 2024-03-30T11:36:13.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:13.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-03-30T11:36:13.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:13.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:36:13.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-03-30T11:36:13.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:13.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-03-30T11:36:13.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-03-30T11:36:13.440 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673034 2024-03-30T11:36:13.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:36:13.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673034 2024-03-30T11:36:13.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:13.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-03-30T11:36:13.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:13.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:36:13.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-03-30T11:36:13.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-03-30T11:36:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-03-30T11:36:13.727 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509501 2024-03-30T11:36:13.727 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:36:14.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509500 -lt 64424509501 2024-03-30T11:36:14.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:36:15.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:36:15.014 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:36:15.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509501 2024-03-30T11:36:15.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:15.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678022 2024-03-30T11:36:15.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:15.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:36:15.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678022 2024-03-30T11:36:15.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:15.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678022 2024-03-30T11:36:15.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678022' 2024-03-30T11:36:15.312 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678022 2024-03-30T11:36:15.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:36:15.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678022 2024-03-30T11:36:15.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:15.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182450 2024-03-30T11:36:15.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:15.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:36:15.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182450 2024-03-30T11:36:15.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:15.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182450 2024-03-30T11:36:15.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-03-30T11:36:15.599 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182450 2024-03-30T11:36:15.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:36:15.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182450 -lt 107374182450 2024-03-30T11:36:15.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:15.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018926 2024-03-30T11:36:15.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:15.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:36:15.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018926 2024-03-30T11:36:15.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:15.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018926 2024-03-30T11:36:15.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018926' 2024-03-30T11:36:15.883 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 128849018926 2024-03-30T11:36:15.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:36:16.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018927 -lt 128849018926 2024-03-30T11:36:16.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:16.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-03-30T11:36:16.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:16.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:36:16.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-03-30T11:36:16.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:16.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-03-30T11:36:16.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-03-30T11:36:16.169 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855397 2024-03-30T11:36:16.169 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:36:16.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855397 2024-03-30T11:36:16.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:16.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-03-30T11:36:16.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:16.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:36:16.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-03-30T11:36:16.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:16.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-03-30T11:36:16.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-03-30T11:36:16.457 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691873 2024-03-30T11:36:16.457 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:36:16.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691873 -lt 171798691873 2024-03-30T11:36:16.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:16.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528344 2024-03-30T11:36:16.743 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:16.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:36:16.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528344 2024-03-30T11:36:16.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:16.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528344 2024-03-30T11:36:16.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-03-30T11:36:16.745 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528344 2024-03-30T11:36:16.746 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:36:17.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528344 -lt 193273528344 2024-03-30T11:36:17.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:17.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-03-30T11:36:17.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:17.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:36:17.051 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-03-30T11:36:17.051 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:17.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:36:17.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-03-30T11:36:17.052 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 330712481794 2024-03-30T11:36:17.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:36:17.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-30T11:36:17.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:36:17.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:17.333 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:17.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:36:17.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:36:17.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:36:17.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:36:17.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:36:17.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:36:17.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:36:17.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:36:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:36:18.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:36:18.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:18.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:18.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:36:18.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:36:18.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:36:18.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:36:18.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:18.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:36:18.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:36:18.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-30T11:36:20.641 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:36:21.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:36:21.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:36:21.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:36:21.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:36:21.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:36:21.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:36:21.160 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:36:21.161 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.5 2024-03-30T11:36:21.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:36:21.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:36:21.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:36:21.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:36:21.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:36:21.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:36:21.189 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:21.196+0000 7f7063814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:21.191 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:21.198+0000 7f7063814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:21.192 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:21.200+0000 7f7063814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:21.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:21.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:36:21.711 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:21.718+0000 7f7063814640 -1 Falling back to public interface 2024-03-30T11:36:21.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:22.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:22.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:22.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:36:22.744 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:36:22.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:22.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:36:23.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:24.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:24.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:24.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:36:24.058 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:36:24.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:24.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:36:24.124 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:24.119+0000 7f7063814640 -1 osd.5 79 log_to_monitors true 2024-03-30T11:36:24.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:25.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:25.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:25.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:36:25.347 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:36:25.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:25.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/2953141002,v1:127.0.0.1:6843/2953141002] [v2:127.0.0.1:6844/2953141002,v1:127.0.0.1:6845/2953141002] exists,up 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:36:25.707 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:36:25.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:36:25.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:36:25.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:36:25.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:36:25.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:36:25.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:36:25.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:36:25.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:36:25.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:36:26.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:36:26.186 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:36:26.186 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:36:26.186 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:36:26.186 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:36:26.186 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.187 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:36:26.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836562 2024-03-30T11:36:26.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836562 2024-03-30T11:36:26.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562' 2024-03-30T11:36:26.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:36:26.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-03-30T11:36:26.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-03-30T11:36:26.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038' 2024-03-30T11:36:26.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:36:26.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509505 2024-03-30T11:36:26.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509505 2024-03-30T11:36:26.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505' 2024-03-30T11:36:26.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:36:26.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678026 2024-03-30T11:36:26.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678026 2024-03-30T11:36:26.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026' 2024-03-30T11:36:26.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:36:26.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182454 2024-03-30T11:36:26.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182454 2024-03-30T11:36:26.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026 4-107374182454' 2024-03-30T11:36:26.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:36:26.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-03-30T11:36:26.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-03-30T11:36:26.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026 4-107374182454 5-356482285570' 2024-03-30T11:36:26.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:26.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:36:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855400 2024-03-30T11:36:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855400 2024-03-30T11:36:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026 4-107374182454 5-356482285570 6-150323855400' 2024-03-30T11:36:27.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:27.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:36:27.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691877 2024-03-30T11:36:27.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691877 2024-03-30T11:36:27.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026 4-107374182454 5-356482285570 6-150323855400 7-171798691877' 2024-03-30T11:36:27.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:27.169 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:36:27.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528348 2024-03-30T11:36:27.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528348 2024-03-30T11:36:27.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026 4-107374182454 5-356482285570 6-150323855400 7-171798691877 8-193273528348' 2024-03-30T11:36:27.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:27.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:36:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-03-30T11:36:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-03-30T11:36:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509505 3-304942678026 4-107374182454 5-356482285570 6-150323855400 7-171798691877 8-193273528348 9-330712481798' 2024-03-30T11:36:27.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:27.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836562 2024-03-30T11:36:27.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:27.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:36:27.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836562 2024-03-30T11:36:27.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:27.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836562 2024-03-30T11:36:27.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-03-30T11:36:27.383 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836562 2024-03-30T11:36:27.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:36:27.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836562 2024-03-30T11:36:27.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:36:28.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:36:28.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:36:28.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836562 2024-03-30T11:36:28.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:28.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-03-30T11:36:28.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:28.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:36:28.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-03-30T11:36:28.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:28.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-03-30T11:36:28.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-03-30T11:36:28.965 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673038 2024-03-30T11:36:28.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:36:29.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673038 2024-03-30T11:36:29.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:29.251 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509505 2024-03-30T11:36:29.251 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:29.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:36:29.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509505 2024-03-30T11:36:29.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:29.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509505 2024-03-30T11:36:29.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-03-30T11:36:29.254 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509505 2024-03-30T11:36:29.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:36:29.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509505 2024-03-30T11:36:29.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:29.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678026 2024-03-30T11:36:29.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:29.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:36:29.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678026 2024-03-30T11:36:29.537 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:29.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678026 2024-03-30T11:36:29.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678026' 2024-03-30T11:36:29.538 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678026 2024-03-30T11:36:29.538 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:36:29.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678027 -lt 304942678026 2024-03-30T11:36:29.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:29.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182454 2024-03-30T11:36:29.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:29.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:36:29.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182454 2024-03-30T11:36:29.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:29.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182454 2024-03-30T11:36:29.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-03-30T11:36:29.824 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182454 2024-03-30T11:36:29.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:36:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182454 2024-03-30T11:36:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:30.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285570 2024-03-30T11:36:30.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:36:30.112 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285570 2024-03-30T11:36:30.112 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:30.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-03-30T11:36:30.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-03-30T11:36:30.113 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 356482285570 2024-03-30T11:36:30.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:36:30.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285570 2024-03-30T11:36:30.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:30.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855400 2024-03-30T11:36:30.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:30.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:36:30.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855400 2024-03-30T11:36:30.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:30.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855400 2024-03-30T11:36:30.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-03-30T11:36:30.399 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855400 2024-03-30T11:36:30.399 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:36:30.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855400 2024-03-30T11:36:30.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:30.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691877 2024-03-30T11:36:30.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:30.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:36:30.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691877 2024-03-30T11:36:30.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:30.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691877 2024-03-30T11:36:30.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-03-30T11:36:30.685 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691877 2024-03-30T11:36:30.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:36:30.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691877 -lt 171798691877 2024-03-30T11:36:30.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:30.997 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528348 2024-03-30T11:36:30.997 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:30.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:36:30.998 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528348 2024-03-30T11:36:30.998 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:30.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528348 2024-03-30T11:36:30.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-03-30T11:36:30.999 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528348 2024-03-30T11:36:30.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:36:31.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528348 -lt 193273528348 2024-03-30T11:36:31.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:31.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-03-30T11:36:31.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:31.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:36:31.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-03-30T11:36:31.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:31.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-03-30T11:36:31.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-03-30T11:36:31.287 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 330712481798 2024-03-30T11:36:31.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:36:31.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-03-30T11:36:31.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:36:31.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:31.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:36:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:36:32.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:36:32.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:36:32.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:32.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:32.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:36:32.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:36:32.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:36:32.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:36:32.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:36:32.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:36:32.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:36:32.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:32.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:33.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:33.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:36:33.390 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:36:33.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:33.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:36:33.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:33.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:36:33.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:36:33.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:33.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:33.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:33.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:33.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:33.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:33.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:02.588611+0000 '>' 2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:33.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:36:34.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:36:34.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:34.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:34.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:34.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:34.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:34.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:34.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:02.588611+0000 '>' 2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:34.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:36:35.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:36:35.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:35.973 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:35.973 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:35.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:35.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:35.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:36.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:02.588611+0000 '>' 2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:36.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:36:37.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:36:37.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:37.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:37.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:37.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:37.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:37.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:02.588611+0000 '>' 2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:37.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:36:38.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:36:38.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:36:38.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:36:38.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:36:38.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:36:38.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:36:38.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:36:38.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:34.359453+0000 '>' 2024-03-30T11:36:02.588611+0000 2024-03-30T11:36:38.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:36:38.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:36:38.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:36:38.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:36:38.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:36:38.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:36:39.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:36:39.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:36:39.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:36:39.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:36:39.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:36:39.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:36:39.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:36:39.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-30T11:36:40.941 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:36:40.941 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:36:40.941 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:36:41.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:36:41.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:36:41.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:36:41.215 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.9 2024-03-30T11:36:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:36:41.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:36:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:36:41.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:36:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:36:41.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:36:41.243 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:41.250+0000 7f5aeb815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:41.245 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:41.252+0000 7f5aeb815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:41.253+0000 7f5aeb815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:41.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:41.765 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:41.772+0000 7f5aeb815640 -1 Falling back to public interface 2024-03-30T11:36:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:36:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:36:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:43.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:44.027 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:36:44.034+0000 7f5aeb815640 -1 osd.9 84 log_to_monitors true 2024-03-30T11:36:44.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:44.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:44.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:36:44.090 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:36:44.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:44.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:44.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:36:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:36:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:36:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:36:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:36:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:36:45.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/1109304279,v1:127.0.0.1:6875/1109304279] [v2:127.0.0.1:6876/1109304279,v1:127.0.0.1:6877/1109304279] exists,up 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:36:45.862 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:36:45.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:36:45.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:36:45.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:36:45.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:36:45.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:36:45.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:36:45.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:36:45.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:36:45.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:36:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:36:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:36:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:36:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:36:46.474 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:46.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:36:46.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836567 2024-03-30T11:36:46.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836567 2024-03-30T11:36:46.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567' 2024-03-30T11:36:46.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:46.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:36:46.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-30T11:36:46.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-30T11:36:46.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043' 2024-03-30T11:36:46.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:46.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:36:46.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-03-30T11:36:46.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-03-30T11:36:46.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510' 2024-03-30T11:36:46.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:46.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:36:47.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678031 2024-03-30T11:36:47.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678031 2024-03-30T11:36:47.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031' 2024-03-30T11:36:47.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:47.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:36:47.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182459 2024-03-30T11:36:47.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182459 2024-03-30T11:36:47.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031 4-107374182459' 2024-03-30T11:36:47.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:47.191 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:36:47.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285575 2024-03-30T11:36:47.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285575 2024-03-30T11:36:47.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031 4-107374182459 5-356482285575' 2024-03-30T11:36:47.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:47.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:36:47.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855406 2024-03-30T11:36:47.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855406 2024-03-30T11:36:47.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031 4-107374182459 5-356482285575 6-150323855406' 2024-03-30T11:36:47.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:47.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:36:47.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-03-30T11:36:47.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-03-30T11:36:47.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031 4-107374182459 5-356482285575 6-150323855406 7-171798691882' 2024-03-30T11:36:47.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:47.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:36:47.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528353 2024-03-30T11:36:47.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528353 2024-03-30T11:36:47.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031 4-107374182459 5-356482285575 6-150323855406 7-171798691882 8-193273528353' 2024-03-30T11:36:47.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:36:47.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:36:47.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-30T11:36:47.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-30T11:36:47.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509510 3-304942678031 4-107374182459 5-356482285575 6-150323855406 7-171798691882 8-193273528353 9-377957122050' 2024-03-30T11:36:47.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:47.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836567 2024-03-30T11:36:47.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:47.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:36:47.718 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836567 2024-03-30T11:36:47.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:47.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836567 2024-03-30T11:36:47.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-03-30T11:36:47.720 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836567 2024-03-30T11:36:47.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:36:48.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836567 2024-03-30T11:36:48.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:36:49.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:36:49.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:36:49.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836567 2024-03-30T11:36:49.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:49.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-30T11:36:49.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:49.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:36:49.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-30T11:36:49.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:49.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-30T11:36:49.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-30T11:36:49.302 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673043 2024-03-30T11:36:49.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:36:49.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-03-30T11:36:49.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:49.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-03-30T11:36:49.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:49.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:36:49.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-03-30T11:36:49.589 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:49.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-03-30T11:36:49.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-03-30T11:36:49.590 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509510 2024-03-30T11:36:49.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:36:49.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509510 2024-03-30T11:36:49.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:49.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678031 2024-03-30T11:36:49.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:49.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:36:49.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678031 2024-03-30T11:36:49.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:49.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678031 2024-03-30T11:36:49.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678031' 2024-03-30T11:36:49.875 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678031 2024-03-30T11:36:49.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:36:50.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678032 -lt 304942678031 2024-03-30T11:36:50.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:50.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182459 2024-03-30T11:36:50.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:50.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:36:50.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182459 2024-03-30T11:36:50.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:50.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182459 2024-03-30T11:36:50.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-03-30T11:36:50.165 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182459 2024-03-30T11:36:50.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:36:50.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182459 2024-03-30T11:36:50.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:50.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285575 2024-03-30T11:36:50.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:50.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:36:50.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285575 2024-03-30T11:36:50.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:50.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285575 2024-03-30T11:36:50.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285575' 2024-03-30T11:36:50.448 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 356482285575 2024-03-30T11:36:50.448 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:36:50.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285576 -lt 356482285575 2024-03-30T11:36:50.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:50.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855406 2024-03-30T11:36:50.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:50.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:36:50.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855406 2024-03-30T11:36:50.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:50.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855406 2024-03-30T11:36:50.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855406' 2024-03-30T11:36:50.733 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855406 2024-03-30T11:36:50.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:36:51.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855406 2024-03-30T11:36:51.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:51.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-03-30T11:36:51.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:51.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:36:51.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-03-30T11:36:51.027 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:51.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-03-30T11:36:51.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-03-30T11:36:51.029 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691882 2024-03-30T11:36:51.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:36:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691882 -lt 171798691882 2024-03-30T11:36:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:51.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528353 2024-03-30T11:36:51.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:51.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:36:51.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528353 2024-03-30T11:36:51.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528353 2024-03-30T11:36:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2024-03-30T11:36:51.314 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528353 2024-03-30T11:36:51.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:36:51.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528353 -lt 193273528353 2024-03-30T11:36:51.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:36:51.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-03-30T11:36:51.598 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:36:51.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:36:51.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-03-30T11:36:51.599 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:36:51.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-30T11:36:51.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-03-30T11:36:51.600 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 377957122050 2024-03-30T11:36:51.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:36:51.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-03-30T11:36:51.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:36:51.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:51.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:52.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:36:52.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:36:52.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:36:52.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:36:52.236 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:36:52.236 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:36:52.236 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:36:52.236 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:36:52.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:36:52.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:36:52.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:36:52.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:36:52.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:36:52.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:36:52.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:36:52.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:36:52.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:36:52.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 228522"' 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 228522' 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 228524"' 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 228524' 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/228525: /' 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 228522 228524' 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 228522 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:36:52.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/228527: /' 2024-03-30T11:36:56.172 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:36:56.173 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:36:56.174 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:36:56.175 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:36:56.175 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:36:56.175 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:36:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:36:56.177 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:36:56.178 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:36:56.178 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:36:56.178 INFO:tasks.workunit.client.0.smithi003.stderr:228527: start osd.9 2024-03-30T11:36:56.178 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:36:56.212 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.212 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:36:56.212 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:36:56.212 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:36:56.212 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:36:56.213 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:36:56.214 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:36:56.215 INFO:tasks.workunit.client.0.smithi003.stderr:228525: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:36:56.216 INFO:tasks.workunit.client.0.smithi003.stderr:228525: start osd.5 2024-03-30T11:36:56.217 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2024-03-30T11:36:56.210+0000 7f1e5d28b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2024-03-30T11:36:56.212+0000 7f1e5d28b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2024-03-30T11:36:56.213+0000 7f1e5d28b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:00.733 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 0 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2024-03-30T11:36:56.728+0000 7f1e5d28b640 -1 Falling back to public interface 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 1 2024-03-30T11:37:00.734 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2024-03-30T11:36:59.461+0000 7f1e5d28b640 -1 osd.9 90 log_to_monitors true 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 3 2024-03-30T11:37:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.736 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:00.752 INFO:tasks.workunit.client.0.smithi003.stderr:228527: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 91 last_c228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:37:00.752 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:37:00.752 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:00.752 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2024-03-30T11:36:56.246+0000 7fedcf973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2024-03-30T11:36:56.249+0000 7fedcf973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2024-03-30T11:36:56.250+0000 7fedcf973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 0 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.753 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2024-03-30T11:36:56.764+0000 7fedcf973640 -1 Falling back to public interface 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 1 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2 2024-03-30T11:37:00.754 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2024-03-30T11:36:59.591+0000 7fedcf973640 -1 osd.5 90 log_to_monitors true 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 3 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:00.755 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:01.802 INFO:tasks.workunit.client.0.smithi003.stderr:228525: osd.5 up in weight 1 up_from 96 up_thru 83 down_at 91 last_clean_interval [83,90) [v2:127.0.0.1:6874/3776960112,v1:127.0.0.1:6875/3776960112] [v2:127.0.0.1:6876/3776960112,v1:127.0.0.1:6877/3776960112] exists,up 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:37:01.802 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:01.802 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:01.802 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:01.803 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 1 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 2 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 3 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 4 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 5 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 6 2024-03-30T11:37:01.804 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 7 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 8 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: 9' 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-03-30T11:37:01.805 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047' 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qlean_interval [88,90) [v2:127.0.0.1:6842/985280851,v1:127.0.0.1:6843/985280851] [v2:127.0.0.1:6844/985280851,v1:127.0.0.1:6845/985280851] exists,up 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:01.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:01.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:01.808 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 1 2024-03-30T11:37:01.808 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 2 2024-03-30T11:37:01.808 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 3 2024-03-30T11:37:01.808 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 4 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 5 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 6 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 7 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 8 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: 9' 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:01.809 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048' 2024-03-30T11:37:01.810 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515' 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678036 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678036 2024-03-30T11:37:02.805 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036' 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036 4-107374182463' 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036 4-107374182463 5-412316860419' 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.806 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036 4-107374182463 5-412316860419 6-150323855410' 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691886 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036 4-107374182463 5-412316860419 6-150323855410 7-171798691886' 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.807 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.clia/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514' 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678035 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678035 2024-03-30T11:37:02.808 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035' 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035 4-107374182464' 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035 4-107374182464 5-412316860418' 2024-03-30T11:37:02.809 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855411 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855411 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035 4-107374182464 5-412316860418 6-150323855411' 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035 4-107374182464 5-412316860418 6-150323855411 7-171798691887' 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:02.810 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:37:03.524 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-03-30T11:37:03.524 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-03-30T11:37:03.524 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035 4-107374182464 5-412316860418 6-150323855411 7-171798691887 8-193273528358' 2024-03-30T11:37:03.524 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509514 3-304942678035 4-107374182464 5-412316860418 6-150323855411 7-171798691887 8-193273528358 9-408021893123' 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.0 seq 21474836572 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:37:03.525 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.1 seq 42949673047 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673047 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:03.526 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.ent.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528357 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528357 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036 4-107374182463 5-412316860419 6-150323855410 7-171798691886 8-193273528357' 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673048 2-64424509515 3-304942678036 4-107374182463 5-412316860419 6-150323855410 7-171798691886 8-193273528357 9-408021893122' 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:37:03.608 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.0 seq 21474836571 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836571 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.1 seq 42949673048 2024-03-30T11:37:03.609 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:37:03.610 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673048 2024-03-30T11:37:03.610 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:03.610 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-03-30T11:37:03.610 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:03.610 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:37:04.424 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.clienclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-03-30T11:37:04.424 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.424 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-03-30T11:37:04.424 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-03-30T11:37:04.424 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.2 seq 64424509514 2024-03-30T11:37:04.424 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509514 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678035 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678035 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678035 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678035' 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.3 seq 304942678035 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678035 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:04.425 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.4 seq 107374182464 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182464 2024-03-30T11:37:04.426 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:04.427 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860418 2024-03-30T11:37:04.427 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:04.427 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:37:04.427 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860418 2024-03-30T11:37:04.427 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.427 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitt.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.2 seq 64424509515 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:04.531 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509515 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678036 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678036 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678036 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678036' 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.3 seq 304942678036 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678036 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:04.532 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.4 seq 107374182463 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182463 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860419 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860419 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:04.533 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting oing osd.5 seq 412316860418' 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.5 seq 412316860418 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855411 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:05.823 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855411 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855411 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.6 seq 150323855411 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855411 -lt 150323855411 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-03-30T11:37:05.824 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.7 seq 171798691887 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691887 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.8 seq 193273528358 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:37:05.825 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528358 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_psd.5 seq 412316860419' 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.5 seq 412316860419 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:37:05.918 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.6 seq 150323855410 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855411 -lt 150323855410 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-03-30T11:37:05.919 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.7 seq 171798691886 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691886 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528357 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528357 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528357 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.8 seq 193273528357 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:37:05.920 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528357 2024-03-30T11:37:07.123 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stag_stats: for s in $seqs 2024-03-30T11:37:07.123 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893123 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893123 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: waiting osd.9 seq 408021893123 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893123 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:37:07.124 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:228525: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:37:07.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:37:07.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 228524 2024-03-30T11:37:07.224 INFO:tasks.workunit.client.0.smithi003.stderr:ts: for s in $seqs 2024-03-30T11:37:07.224 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893122 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893122 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: waiting osd.9 seq 408021893122 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893122 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:37:07.225 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:228527: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:37:07.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:37:07.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:07.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:07.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:07.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:37:08.055 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:37:08.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:08.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:37:08.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:08.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:37:08.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:37:08.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:08.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:08.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:08.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:08.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:08.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:08.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:34.359453+0000 '>' 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:08.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:09.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:09.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:09.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:09.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:09.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:09.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:09.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:09.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:34.359453+0000 '>' 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:09.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:10.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:10.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:10.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:10.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:10.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:10.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:10.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:10.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:34.359453+0000 '>' 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:10.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:11.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:11.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:11.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:11.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:11.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:11.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:11.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:12.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:34.359453+0000 '>' 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:12.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:13.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:13.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:13.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:13.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:13.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:13.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:13.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:13.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:36:34.359453+0000 '>' 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:13.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:14.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:14.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:14.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:14.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:14.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:14.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:14.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:14.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:37:08.495597+0000 '>' 2024-03-30T11:36:34.359453+0000 2024-03-30T11:37:14.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:37:14.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:37:14.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:14.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:37:14.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 231953"' 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 231953' 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 231955"' 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 231955' 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 231953 231955' 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 231953 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/231956: /' 2024-03-30T11:37:14.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:37:14.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/231958: /' 2024-03-30T11:37:17.923 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:37:17.924 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:37:17.924 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:37:17.924 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:37:17.924 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:17.924 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:37:17.925 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:17.926 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:37:17.926 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:37:17.926 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:37:17.926 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:37:17.926 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:37:17.927 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-30T11:37:17.927 INFO:tasks.workunit.client.0.smithi003.stderr:231956: _ 2024-03-30T11:37:17.927 INFO:tasks.workunit.client.0.smithi003.stderr:231956: hinfo_key 2024-03-30T11:37:17.927 INFO:tasks.workunit.client.0.smithi003.stderr:231956: snapset 2024-03-30T11:37:17.927 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:37:17.928 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:37:17.928 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:37:17.928 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:37:17.928 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:37:17.928 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:37:17.928 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:37:17.929 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:37:17.929 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:37:17.929 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:37:17.930 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:37:17.931 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:37:17.932 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:37:17.932 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:37:17.932 INFO:tasks.workunit.client.0.smithi003.stderr:231956: start osd.5 2024-03-30T11:37:18.209 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:37:18.210 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-30T11:37:18.211 INFO:tasks.workunit.client.0.smithi003.stderr:231958: _ 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: hinfo_key 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: snapset 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:37:18.212 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:37:18.213 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:37:18.214 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:37:18.215 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:37:18.215 INFO:tasks.workunit.client.0.smithi003.stderr:231958: start osd.9 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2024-03-30T11:37:17.963+0000 7f84aaa1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:22.540 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2024-03-30T11:37:17.965+0000 7f84aaa1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2024-03-30T11:37:17.967+0000 7f84aaa1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 0 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2024-03-30T11:37:18.479+0000 7f84aaa1f640 -1 Falling back to public interface 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.541 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 1 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2024-03-30T11:37:20.885+0000 7f84aaa1f640 -1 osd.5 97 log_to_monitors true 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.542 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:22.543 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 3 2024-03-30T11:37:22.543 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.543 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:22.764 INFO:tasks.workunit.client.0.smithi003.stderr:231956: osd.5 up in weight 1 up_from 102 up_thru 102 downovery_ops 2024-03-30T11:37:22.764 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2024-03-30T11:37:18.246+0000 7f4496c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2024-03-30T11:37:18.267+0000 7f4496c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2024-03-30T11:37:18.268+0000 7f4496c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.765 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 0 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2024-03-30T11:37:18.783+0000 7f4496c0d640 -1 Falling back to public interface 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 1 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:22.766 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2024-03-30T11:37:21.599+0000 7f4496c0d640 -1 osd.9 97 log_to_monitors true 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 3 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:22.767 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:23.709 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/c_at 98 last_clean_interval [96,97) [v2:127.0.0.1:6842/1237455654,v1:127.0.0.1:6843/1237455654] [v2:127.0.0.1:6844/1237455654,v1:127.0.0.1:6845/1237455654] exists,up 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:23.710 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 1 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 2 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 3 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 4 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 5 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 6 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 7 2024-03-30T11:37:23.711 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 8 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: 9' 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053' 2024-03-30T11:37:23.712 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:24.726 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/cloneph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:24.726 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:24.726 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:24.726 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:37:24.726 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 4 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: osd.9 up in weight 1 up_from 105 up_thru 0 down_at 98 last_clean_interval [95,97) [v2:127.0.0.1:6874/3350461808,v1:127.0.0.1:6875/3350461808] [v2:127.0.0.1:6876/3350461808,v1:127.0.0.1:6877/3350461808] exists,up 0cddbc61-5758-4439-8e42-fa1b67b5ceae 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:24.727 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 1 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 2 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 3 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 4 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 5 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 6 2024-03-30T11:37:24.728 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 7 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 8 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: 9' 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-03-30T11:37:24.729 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-03-30T11:37:25.067 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ide.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:37:25.067 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521' 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678042 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678042 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042' 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042 4-107374182469' 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:37:25.068 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664194 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664194 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042 4-107374182469 5-438086664194' 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855416 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855416 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042 4-107374182469 5-438086664194 6-150323855416' 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042 4-107374182469 5-438086664194 6-150323855416 7-171798691892' 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.069 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:37:25.659 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cs 2024-03-30T11:37:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-03-30T11:37:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-03-30T11:37:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054' 2024-03-30T11:37:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.660 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522' 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678043 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678043 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043' 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-03-30T11:37:25.661 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043 4-107374182471' 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664195 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664195 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043 4-107374182471 5-438086664195' 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043 4-107374182471 5-438086664195 6-150323855417' 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.662 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:37:25.848 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528363 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528363 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042 4-107374182469 5-438086664194 6-150323855416 7-171798691892 8-193273528363' 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509521 3-304942678042 4-107374182469 5-438086664194 6-150323855416 7-171798691892 8-193273528363 9-450971566082' 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:25.849 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.0 seq 21474836577 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836577 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:25.850 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.1 seq 42949673053 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673053 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-03-30T11:37:25.851 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:25.852 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:37:26.199 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cep202: flush_pg_stats: seq=171798691893 2024-03-30T11:37:26.199 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-03-30T11:37:26.199 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043 4-107374182471 5-438086664195 6-150323855417 7-171798691893' 2024-03-30T11:37:26.199 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:26.199 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:37:26.199 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043 4-107374182471 5-438086664195 6-150323855417 7-171798691893 8-193273528364' 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673054 2-64424509522 3-304942678043 4-107374182471 5-438086664195 6-150323855417 7-171798691893 8-193273528364 9-450971566083' 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-03-30T11:37:26.200 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.0 seq 21474836579 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-03-30T11:37:26.201 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-03-30T11:37:26.202 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.1 seq 42949673054 2024-03-30T11:37:26.721 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214:htest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-03-30T11:37:26.721 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:26.721 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-03-30T11:37:26.721 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-03-30T11:37:26.721 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.2 seq 64424509521 2024-03-30T11:37:26.721 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678042 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678042 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678042 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.3 seq 304942678042 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678042 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:26.722 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.4 seq 107374182469 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182469 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-438086664194 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:37:26.723 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-438086664194 2024-03-30T11:37:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:26.724 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664194 2024-03-30T11:37:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:37:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673054 2024-03-30T11:37:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-03-30T11:37:27.462 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.2 seq 64424509522 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509522 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678043 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678043 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678043 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.3 seq 304942678043 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678043 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.4 seq 107374182471 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:37:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182471 2024-03-30T11:37:27.465 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.465 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-438086664195 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //hom echo 'waiting osd.5 seq 438086664194' 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.5 seq 438086664194 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664195 -lt 438086664194 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855416 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855416 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855416 2024-03-30T11:37:27.983 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.6 seq 150323855416 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855416 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.7 seq 171798691892 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691892 2024-03-30T11:37:27.984 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528363 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528363 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528363 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.8 seq 193273528363 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:37:27.985 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528364 -lt 193273528363 2024-03-30T11:37:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:37:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-438086664195 2024-03-30T11:37:28.370 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664195 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 438086664195' 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.5 seq 438086664195 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664195 -lt 438086664195 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-03-30T11:37:28.371 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.6 seq 150323855417 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855417 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.7 seq 171798691893 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691894 -lt 171798691893 2024-03-30T11:37:28.372 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:28.373 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-03-30T11:37:28.373 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:28.373 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:37:28.373 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-03-30T11:37:29.387 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_p210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566082 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566082 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: waiting osd.9 seq 450971566082 2024-03-30T11:37:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566082 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:29.389 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:37:29.390 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:37:29.391 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:37:29.391 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:37:29.391 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:37:29.391 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:37:29.391 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:29.391 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:30.002 INFO:tasks.workunit.client.0.smithi003.stderr:23195g_stats: cut -d - -f 2 2024-03-30T11:37:30.002 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-03-30T11:37:30.002 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-03-30T11:37:30.002 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.8 seq 193273528364 2024-03-30T11:37:30.002 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528364 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566083 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566083 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: waiting osd.9 seq 450971566083 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566083 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T11:37:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/u6: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:30.631 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:37:30.632 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:37:30.633 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:37:30.633 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:30.633 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:30.633 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:30.633 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:30.633 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:30.784 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:231958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:37:30.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:json pg dump pgs 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:231956: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:231956: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 231955 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:37:31.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:37:31.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:37:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 235381"' 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 235381' 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 235383"' 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 235383' 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 235381 235383' 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:37:31.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 235381 2024-03-30T11:37:31.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/235384: /' 2024-03-30T11:37:31.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:37:31.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/235386: /' 2024-03-30T11:37:34.256 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.256 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:37:34.256 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:37:34.257 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:37:34.258 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:37:34.259 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:37:34.260 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:37:34.261 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:37:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:37:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:37:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:37:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:235386: start osd.5 2024-03-30T11:37:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:37:34.358 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:37:34.359 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:37:34.360 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:37:34.361 INFO:tasks.workunit.client.0.smithi003.stderr:235384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:37:34.362 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:37:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:37:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:37:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:37:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:37:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:235384: start osd.3 2024-03-30T11:37:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:37:38.764 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:37:38.764 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:37:38.764 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:38.764 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2024-03-30T11:37:34.296+0000 7f9a5da19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2024-03-30T11:37:34.298+0000 7f9a5da19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2024-03-30T11:37:34.300+0000 7f9a5da19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 0 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:38.765 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2024-03-30T11:37:34.830+0000 7f9a5da19640 -1 Falling back to public interface 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 1 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2 2024-03-30T11:37:38.766 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2024-03-30T11:37:37.477+0000 7f9a5da19640 -1 osd.5 106 log_to_monitors true 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 3 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:39.086 INFO:tasks.workunit.client.0.smithi003.stderr:235386: osd.5 up in weight 1 up_from 111 up_thru 102 down_at 107 l235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2024-03-30T11:37:34.394+0000 7f4781a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2024-03-30T11:37:34.396+0000 7f4781a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2024-03-30T11:37:34.398+0000 7f4781a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:39.087 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 0 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2024-03-30T11:37:34.924+0000 7f4781a0d640 -1 Falling back to public interface 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 1 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2024-03-30T11:37:37.703+0000 7f4781a0d640 -1 osd.3 106 log_to_monitors true 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 3 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235384: osd.3 up in weight 1 up_from 111 up_thru 111 down_at 107 last_clean_interval [102,106) [v2:127.0.0.1:6826/1126496445,v1:127.0.0.1:6827/1126496445] [v2:127.0.0.1:6828/1126496445,v1:127.0.0.1:6829/1126496445] exists,up 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:39.813 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 1 2024-03-30T11:37:39.814 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 2 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 3 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 4 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 5 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 6 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 7 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 8 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: 9' 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836583 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836583 2024-03-30T11:37:39.815 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583' 2024-03-30T11:37:39.816 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:39.816 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:39.816 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-03-30T11:37:39.816 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-03-30T11:37:39.816 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058' 2024-03-30T11:37:39.816 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.clieast_clean_interval [71,106) [v2:127.0.0.1:6842/3216698879,v1:127.0.0.1:6843/3216698879] [v2:127.0.0.1:6844/3216698879,v1:127.0.0.1:6845/3216698879] exists,up 05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:39.967 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 1 2024-03-30T11:37:39.968 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 2 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 3 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 4 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 5 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 6 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 7 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 8 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: 9' 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-03-30T11:37:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-03-30T11:37:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059' 2024-03-30T11:37:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.770 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.cliennt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:37:40.770 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-03-30T11:37:40.770 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-03-30T11:37:40.770 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526' 2024-03-30T11:37:40.770 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.770 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858' 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182475 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182475 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858 4-107374182475' 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858 4-107374182475 5-476741369858' 2024-03-30T11:37:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858 4-107374182475 5-476741369858 6-150323855421' 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858 4-107374182475 5-476741369858 6-150323855421 7-171798691897' 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.772 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:37:40.881 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:37:40.881 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-03-30T11:37:40.881 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-03-30T11:37:40.881 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527' 2024-03-30T11:37:40.881 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859' 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859 4-107374182476' 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-03-30T11:37:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859 4-107374182476 5-476741369859' 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855422 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855422 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859 4-107374182476 5-476741369859 6-150323855422' 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859 4-107374182476 5-476741369859 6-150323855422 7-171798691898' 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtestt/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858 4-107374182475 5-476741369858 6-150323855421 7-171798691897 8-193273528368' 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566087 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566087 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673058 2-64424509526 3-476741369858 4-107374182475 5-476741369858 6-150323855421 7-171798691897 8-193273528368 9-450971566087' 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836583 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836583 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836583 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.0 seq 21474836583 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836584 -lt 21474836583 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:41.501 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.1 seq 42949673058 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673058 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:41.502 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:37:41.581 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest//clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-30T11:37:41.581 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-30T11:37:41.581 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859 4-107374182476 5-476741369859 6-150323855422 7-171798691898 8-193273528369' 2024-03-30T11:37:41.581 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:41.581 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509527 3-476741369859 4-107374182476 5-476741369859 6-150323855422 7-171798691898 8-193273528369 9-450971566088' 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.0 seq 21474836584 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836584 -lt 21474836584 2024-03-30T11:37:41.582 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.1 seq 42949673059 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:41.583 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/cclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.2 seq 64424509526 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509526 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:42.522 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369858 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369858 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.3 seq 476741369858 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369858 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182475 2024-03-30T11:37:42.523 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182475 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182475 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.4 seq 107374182475 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182475 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369858 2024-03-30T11:37:42.524 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:42.525 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:37:42.525 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369858 2024-03-30T11:37:42.525 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:42.525 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-30T11:37:43.706 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 476741369858' 2024-03-30T11:37:43.706 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.5 seq 476741369858 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369858 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.6 seq 150323855421 2024-03-30T11:37:43.707 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855421 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.7 seq 171798691897 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691897 2024-03-30T11:37:43.708 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.8 seq 193273528368 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:37:43.709 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528368 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.2 seq 64424509527 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509527 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:37:43.859 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509527 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369859 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369859 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.3 seq 476741369859 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:37:43.860 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369859 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.4 seq 107374182476 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182476 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:43.861 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-476741369859 2024-03-30T11:37:43.862 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-476741369859 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 476741369859' 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.5 seq 476741369859 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:37:44.743 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369860 -lt 476741369859 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855422 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855422 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855422 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.6 seq 150323855422 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855423 -lt 150323855422 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:44.744 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.7 seq 171798691898 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691899 -lt 171798691898 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:44.745 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:37:44.746 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-30T11:37:44.746 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:44.746 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-30T11:37:45.048 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.cllush_pg_stats: for s in $seqs 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566087 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566087 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566087 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566087' 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: waiting osd.9 seq 450971566087 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566087 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:37:45.049 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:235386: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:37:45.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:ient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.8 seq 193273528369 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566088 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566088 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-03-30T11:37:46.312 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566088' 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: waiting osd.9 seq 450971566088 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566088 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:37:46.313 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:235384: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 235383 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:37:46.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:37:46.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:37:46.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:37:46.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:37:46.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:37:46.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:37:46.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:46.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:46.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:46.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:37:47.145 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:37:47.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:47.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:47.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:47.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:37:08.495597+0000 '>' 2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:47.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:48.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:48.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:48.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:48.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:48.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:48.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:48.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:48.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:37:08.495597+0000 '>' 2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:48.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:49.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:49.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:49.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:49.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:49.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:49.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:49.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:50.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:37:08.495597+0000 '>' 2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:50.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:37:51.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:37:51.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:37:51.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:37:51.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:37:51.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:37:51.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:37:51.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:37:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:37:47.915524+0000 '>' 2024-03-30T11:37:08.495597+0000 2024-03-30T11:37:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:37:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:37:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:37:51.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 238785"' 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 238785' 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 238787"' 2024-03-30T11:37:51.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 238787' 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 238785 238787' 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 238785 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/238788: /' 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:51.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/238790: /' 2024-03-30T11:37:54.219 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:37:54.220 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: _ 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: hinfo_key 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: snapset 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:37:54.221 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.222 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:37:54.222 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:37:54.222 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:37:54.222 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:37:54.222 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:37:54.223 INFO:tasks.workunit.client.0.smithi003.stderr:238788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:37:54.224 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:37:54.225 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:37:54.225 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:37:54.225 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:37:54.225 INFO:tasks.workunit.client.0.smithi003.stderr:238788: start osd.3 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:37:54.699 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: _ 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: hinfo_key 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: snapset 2024-03-30T11:37:54.700 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:37:54.701 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:37:54.702 INFO:tasks.workunit.client.0.smithi003.stderr:238790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:37:54.703 INFO:tasks.workunit.client.0.smithi003.stderr:238790: start osd.5 2024-03-30T11:37:58.429 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2024-03-30T11:37:54.259+0000 7f389f411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2024-03-30T11:37:54.261+0000 7f389f411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2024-03-30T11:37:54.262+0000 7f389f411640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:58.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 0 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2024-03-30T11:37:54.791+0000 7f389f411640 -1 Falling back to public interface 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 1 2024-03-30T11:37:58.431 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2024-03-30T11:37:57.289+0000 7f389f411640 -1 osd.3 112 log_to_monitors true 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:58.432 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:58.433 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 3 2024-03-30T11:37:58.433 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2024-03-30T11:37:54.734+0000 7fa05abfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2024-03-30T11:37:54.737+0000 7fa05abfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2024-03-30T11:37:54.738+0000 7fa05abfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:37:59.262 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 0 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2024-03-30T11:37:55.255+0000 7fa05abfd640 -1 Falling back to public interface 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 1 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:59.263 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2024-03-30T11:37:58.114+0000 7fa05abfd640 -1 osd.5 112 log_to_monitors true 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 3 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:37:59.264 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:37:59.594 INFO:tasks.workunit.client.0.smithi003.stderr:238790: osd.5 up in weight 1 up_from 119 up_thru 111 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:37:59.594 INFO:tasks.workunit.client.0.smithi003.stderr:238788: osd.3 up in weight 1 up_from 117 up_thru 111 down_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6826/3446136668,v1:127.0.0.1:6827/3446136668] [v2:127.0.0.1:6828/3446136668,v1:127.0.0.1:6829/3446136668] exists,up 05caf833-bdd9-4407-a7b3-30c0a63bacdb 2024-03-30T11:37:59.594 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:37:59.594 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:37:59.595 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 1 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 2 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 3 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 4 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 5 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 6 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 7 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 8 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: 9' 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:59.596 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-03-30T11:37:59.597 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064' 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0n_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6842/4144122065,v1:127.0.0.1:6843/4144122065] [v2:127.0.0.1:6844/4144122065,v1:127.0.0.1:6845/4144122065] exists,up 67bce22a-3f3f-4bd8-b894-3ffb47c82ccf 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:38:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:238790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 1 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 2 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 3 2024-03-30T11:38:00.269 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 4 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 5 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 6 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 7 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 8 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: 9' 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836590 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836590 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590' 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673065 2024-03-30T11:38:00.270 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673065 2024-03-30T11:38:00.271 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065' 2024-03-30T11:38:00.271 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.846 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest//qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532' 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634' 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.847 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634 4-107374182481' 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634 4-107374182481 5-511101108226' 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634 4-107374182481 5-511101108226 6-150323855427' 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:00.848 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:38:00.849 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691903 2024-03-30T11:38:00.849 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691903 2024-03-30T11:38:00.849 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634 4-107374182481 5-511101108226 6-150323855427 7-171798691903' 2024-03-30T11:38:00.849 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533' 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173635 2024-03-30T11:38:01.503 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173635 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635' 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182482 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182482 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635 4-107374182482' 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108227 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108227 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635 4-107374182482 5-511101108227' 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855428 2024-03-30T11:38:01.504 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855428 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635 4-107374182482 5-511101108227 6-150323855428' 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691904 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635 4-107374182482 5-511101108227 6-150323855428 7-171798691904' 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.505 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:38:01.787 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuna/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:38:01.787 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-03-30T11:38:01.787 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-03-30T11:38:01.787 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634 4-107374182481 5-511101108226 6-150323855427 7-171798691903 8-193273528374' 2024-03-30T11:38:01.787 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:01.787 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566093 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566093 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673064 2-64424509532 3-502511173634 4-107374182481 5-511101108226 6-150323855427 7-171798691903 8-193273528374 9-450971566093' 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.0 seq 21474836589 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:38:01.788 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836589 -lt 21474836589 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673064 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.1 seq 42949673064 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673064 -lt 42949673064 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-03-30T11:38:01.789 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.2 seq 64424509532 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-502511173634 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-502511173634 2024-03-30T11:38:02.686 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 502511173634' 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.3 seq 502511173634 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173635 -lt 502511173634 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.4 seq 107374182481 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:38:02.687 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182481 2024-03-30T11:38:02.688 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:02.688 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108226 2024-03-30T11:38:02.688 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:02.688 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:38:02.688 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108226 2024-03-30T11:38:02.688 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5111tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635 4-107374182482 5-511101108227 6-150323855428 7-171798691904 8-193273528376' 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673065 2-64424509533 3-502511173635 4-107374182482 5-511101108227 6-150323855428 7-171798691904 8-193273528376 9-450971566094' 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836590 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:03.537 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836590 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836590 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.0 seq 21474836590 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836589 -lt 21474836590 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836590 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673065 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673065 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673065 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.1 seq 42949673065 2024-03-30T11:38:03.539 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:38:03.924 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673066 -lt 429401108226 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 511101108226' 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.5 seq 511101108226 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108226 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-03-30T11:38:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.6 seq 150323855427 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855428 -lt 150323855427 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691903 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691903 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691903 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.7 seq 171798691903 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691903 2024-03-30T11:38:03.926 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.8 seq 193273528374 2024-03-30T11:38:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:38:04.430 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 19673065 2024-03-30T11:38:04.430 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:04.430 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-03-30T11:38:04.430 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.2 seq 64424509533 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509533 -lt 64424509533 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-502511173635 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-502511173635 2024-03-30T11:38:04.431 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173635 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 502511173635' 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.3 seq 502511173635 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173636 -lt 502511173635 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182482 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182482 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182482 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.4 seq 107374182482 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:38:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182482 2024-03-30T11:38:04.433 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:04.433 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-511101108227 2024-03-30T11:38:04.433 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:05.297 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-he93273528376 -lt 193273528374 2024-03-30T11:38:05.297 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:05.297 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566093 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566093 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566093 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566093' 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: waiting osd.9 seq 450971566093 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566093 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:38:05.298 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:38:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:238788: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:38:05.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:38:05.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:38:05.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 238787 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:lpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-511101108227 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108227 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 511101108227' 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.5 seq 511101108227 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108228 -lt 511101108227 2024-03-30T11:38:05.322 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855428 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855428 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855428 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.6 seq 150323855428 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855428 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:05.323 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.7 seq 171798691904 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691905 -lt 171798691904 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:05.324 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-03-30T11:38:06.935 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-03-30T11:38:06.935 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.8 seq 193273528376 2024-03-30T11:38:06.935 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528376 -lt 193273528376 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566094 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566094 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566094' 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: waiting osd.9 seq 450971566094 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566094 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:38:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:238790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:38:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:38:06.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:38:06.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:38:06.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:38:06.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:38:06.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:38:06.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:38:06.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:38:06.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:38:06.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:38:06.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:38:06.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:38:06.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:38:06.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:38:06.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:38:06.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:38:06.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:38:07.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:38:07.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:38:07.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:38:07.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:38:07.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:38:07.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:38:07.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:38:07.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:38:07.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:38:07.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:38:07.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:38:07.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:38:07.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:38:07.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:38:07.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:38:07.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:07.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:07.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:38:07.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:38:07.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:38:07.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:38:07.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:38:07.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:38:07.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:38:07.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:38:07.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:38:07.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:38:07.176 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:38:07.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:38:07.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:38:07.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:38:07.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:38:07.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:38:07.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:38:07.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:38:07.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:38:07.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:38:07.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:38:07.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:38:07.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:07.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:07.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:38:07.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:38:07.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:38:07.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:38:07.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:38:07.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:07.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:07.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:38:07.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:38:07.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:38:07.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:38:07.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:38:07.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:38:07.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:38:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:38:07.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:38:07.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:38:07.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:38:07.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:38:07.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:38:07.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:38:07.754 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:38:07.755 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:38:07.755 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:07.755 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:07.755 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:38:07.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:38:07.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:38:07.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:38:08.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:38:08.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:38:08.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:38:08.649 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:38:08.649 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:08.649 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:08.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:38:08.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:38:08.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:38:08.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:38:08.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:38:08.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:38:08.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:38:08.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79feaf2f-cbc6-4e42-a384-d18bc803dd66 2024-03-30T11:38:08.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79feaf2f-cbc6-4e42-a384-d18bc803dd66' 2024-03-30T11:38:08.684 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 79feaf2f-cbc6-4e42-a384-d18bc803dd66 2024-03-30T11:38:08.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:38:08.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCg+Qdm/RUkKhAAM3TAg6uolmyujdffQRhFYg== 2024-03-30T11:38:08.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCg+Qdm/RUkKhAAM3TAg6uolmyujdffQRhFYg=="}' 2024-03-30T11:38:08.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79feaf2f-cbc6-4e42-a384-d18bc803dd66 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:38:08.904 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:38:08.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:38:08.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCg+Qdm/RUkKhAAM3TAg6uolmyujdffQRhFYg== --osd-uuid 79feaf2f-cbc6-4e42-a384-d18bc803dd66 2024-03-30T11:38:08.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:08.949+0000 7fe3f060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:08.955 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:08.961+0000 7fe3f060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:08.962+0000 7fe3f060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:08.957 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:08.963+0000 7fe3f060f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:38:08.957 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:08.963+0000 7fe3f060f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:38:08.957 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:08.963+0000 7fe3f060f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:38:09.047 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:09.053+0000 7fe3f060f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:38:14.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:38:14.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:38:14.565 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:38:14.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:38:14.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:38:14.933 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:38:14.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:38:14.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:38:14.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:38:14.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:38:14.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:38:14.959 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:14.965+0000 7f8afa011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:14.961 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:14.967+0000 7f8afa011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:14.962 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:14.968+0000 7f8afa011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:15.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:38:15.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:38:15.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:38:15.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:38:15.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:38:15.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:15.236 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:38:15.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:38:15.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:15.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:15.493 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:15.499+0000 7f8afa011640 -1 Falling back to public interface 2024-03-30T11:38:15.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:38:16.517 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:38:16.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:16.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:16.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:17.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:17.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:17.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:38:17.812 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:38:17.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:17.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:17.827 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:17.834+0000 7f8afa011640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:38:18.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:19.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:19.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:19.255 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:38:19.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:38:19.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:19.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:19.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:20.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:20.568 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:38:20.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:20.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:38:20.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:20.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:20.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:21.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:21.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:21.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:38:21.862 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:38:21.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:21.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:22.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:23.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:23.153 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:38:23.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:23.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:38:23.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:23.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:23.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:24.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:24.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:24.542 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:38:24.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:38:24.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:24.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:24.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:25.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:25.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:25.834 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:38:25.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:38:25.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:25.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:26.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:27.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:27.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:27.128 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:38:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:38:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:27.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:28.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:28.421 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:38:28.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:28.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:38:28.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:28.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:29.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:30.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:30.003 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:38:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:38:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:30.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:30.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:31.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:31.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:31.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:38:31.517 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:38:31.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:31.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:31.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:32.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:32.880 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:38:32.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:32.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:38:32.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:32.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:33.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:34.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:34.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:34.600 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:38:34.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:38:34.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:34.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:35.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:36.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:36.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:36.656 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:38:36.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:38:36.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:36.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:37.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:38.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:38.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:38.374 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:38:38.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:38:38.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:38.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:39.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:39.810 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:38:39.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:39.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:38:39.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:39.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:38:40.101 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/956372496,v1:127.0.0.1:6803/956372496] [v2:127.0.0.1:6804/956372496,v1:127.0.0.1:6805/956372496] exists,up 79feaf2f-cbc6-4e42-a384-d18bc803dd66 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:38:40.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:38:40.106 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:38:40.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:38:40.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:38:40.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1215f91f-805c-4e58-97e4-f131092e5701 2024-03-30T11:38:40.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1215f91f-805c-4e58-97e4-f131092e5701' 2024-03-30T11:38:40.108 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 1215f91f-805c-4e58-97e4-f131092e5701 2024-03-30T11:38:40.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:38:40.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDA+QdmhFS5BxAA1rF2C/AZzwj740mAYStn5w== 2024-03-30T11:38:40.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDA+QdmhFS5BxAA1rF2C/AZzwj740mAYStn5w=="}' 2024-03-30T11:38:40.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1215f91f-805c-4e58-97e4-f131092e5701 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:38:40.466 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:38:40.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:38:40.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDA+QdmhFS5BxAA1rF2C/AZzwj740mAYStn5w== --osd-uuid 1215f91f-805c-4e58-97e4-f131092e5701 2024-03-30T11:38:40.507 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.513+0000 7f1a3741a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:40.551 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.558+0000 7f1a3741a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:40.552 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.559+0000 7f1a3741a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:40.552 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.559+0000 7f1a3741a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:38:40.552 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.559+0000 7f1a3741a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:38:40.553 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.559+0000 7f1a3741a640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:38:40.617 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:40.625+0000 7f1a3741a640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:38:46.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:38:46.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:38:46.275 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:38:46.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:38:46.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:38:46.645 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:38:46.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:38:46.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:38:46.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:38:46.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:38:46.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:38:46.671 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:46.678+0000 7fc3ac21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:46.672 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:46.680+0000 7fc3ac21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:46.673 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:46.681+0000 7fc3ac21e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:38:46.931 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:38:46.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:46.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:47.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:47.195+0000 7fc3ac21e640 -1 Falling back to public interface 2024-03-30T11:38:47.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:48.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:48.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:48.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:38:48.218 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:38:48.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:48.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:48.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:49.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:49.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:49.508 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:38:49.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:38:49.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:49.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:49.531 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:38:49.538+0000 7fc3ac21e640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:38:49.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:50.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:50.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:50.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:38:50.849 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:38:50.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:50.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:51.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:52.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:52.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:52.170 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:38:52.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:38:52.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:52.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:52.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:53.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:53.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:53.554 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:38:53.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:38:53.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:53.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:53.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:54.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:54.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:54.850 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:38:54.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:38:54.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:54.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:55.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:56.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:56.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:56.146 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:38:56.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:38:56.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:56.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:56.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:57.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:57.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:57.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:38:57.488 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:38:57.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:57.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:57.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:38:58.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:38:58.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:38:58.827 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:38:58.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:38:58.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:38:58.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:38:59.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:00.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:00.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:00.147 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:39:00.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:39:00.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:00.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:00.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:01.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:01.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:01.441 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:39:01.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:39:01.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:01.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:03.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:03.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:03.082 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:39:03.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:39:03.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:03.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:03.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:04.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:04.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:04.817 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:39:04.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:39:04.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:04.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:05.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:06.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:06.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:06.158 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:39:06.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:39:06.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:06.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:06.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:07.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:07.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:07.846 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:39:07.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:39:07.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:07.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:08.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:09.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:09.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:09.491 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:39:09.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:39:09.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:09.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:10.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:11.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:11.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:11.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:39:11.210 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:39:11.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:11.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:11.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:12.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:12.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:12.513 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:39:12.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:39:12.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:12.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:39:12.802 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1686986966,v1:127.0.0.1:6811/1686986966] [v2:127.0.0.1:6812/1686986966,v1:127.0.0.1:6813/1686986966] exists,up 1215f91f-805c-4e58-97e4-f131092e5701 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:39:12.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:39:12.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:39:12.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0c8408e-42ac-495a-8534-3c15d6720161 2024-03-30T11:39:12.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c0c8408e-42ac-495a-8534-3c15d6720161' 2024-03-30T11:39:12.810 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 c0c8408e-42ac-495a-8534-3c15d6720161 2024-03-30T11:39:12.811 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:39:12.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDg+QdmIMGJMRAAgo6g6p8oST7AvjaKESPGWQ== 2024-03-30T11:39:12.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDg+QdmIMGJMRAAgo6g6p8oST7AvjaKESPGWQ=="}' 2024-03-30T11:39:12.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0c8408e-42ac-495a-8534-3c15d6720161 -i td/osd-scrub-repair/2/new.json 2024-03-30T11:39:13.154 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:39:13.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:39:13.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDg+QdmIMGJMRAAgo6g6p8oST7AvjaKESPGWQ== --osd-uuid c0c8408e-42ac-495a-8534-3c15d6720161 2024-03-30T11:39:13.194 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.201+0000 7fa68519a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:13.222 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.229+0000 7fa68519a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:13.223 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.230+0000 7fa68519a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:13.224 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.231+0000 7fa68519a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:39:13.224 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.231+0000 7fa68519a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:39:13.224 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.231+0000 7fa68519a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:39:13.289 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:13.295+0000 7fa68519a640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:39:19.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:39:19.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:39:19.069 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:39:19.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:39:19.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:39:19.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:39:19.438 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:39:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:39:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:39:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:39:19.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:39:19.465 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:19.472+0000 7fe1a8a17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:19.467 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:19.474+0000 7fe1a8a17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:19.468 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:19.475+0000 7fe1a8a17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:19.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:39:19.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:39:19.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:39:19.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:39:19.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:39:19.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:19.723 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:39:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:39:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:19.983 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:19.990+0000 7fe1a8a17640 -1 Falling back to public interface 2024-03-30T11:39:20.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:21.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:21.043 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:39:21.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:21.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:39:21.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:21.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:21.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:22.304 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:22.310+0000 7fe1a8a17640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:39:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:22.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:39:22.333 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:39:22.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:22.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:22.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:23.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:23.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:23.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:39:23.773 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:39:23.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:23.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:24.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:25.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:25.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:25.078 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:39:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:39:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:25.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:25.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:26.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:26.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:26.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:39:26.371 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:39:26.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:26.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:26.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:27.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:27.667 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:39:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:39:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:27.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:28.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:29.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:29.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:29.003 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:39:29.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:39:29.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:29.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:29.317 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:30.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:30.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:30.319 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:39:30.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:39:30.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:30.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:30.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:31.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:31.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:31.612 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:39:31.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:39:31.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:31.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:32.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:33.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:33.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:33.003 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:39:33.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:39:33.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:33.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:33.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:34.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:34.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:34.304 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:39:34.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:39:34.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:34.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:35.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:36.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:36.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:36.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:39:36.127 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:39:36.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:36.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:36.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:37.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:37.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:37.608 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:39:37.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:39:37.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:37.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:38.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:39.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:39.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:39.376 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:39:39.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:39:39.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:39.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:40.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:41.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:41.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:41.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:39:41.209 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:39:41.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:41.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:41.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:42.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:42.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:42.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:39:42.864 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:39:42.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:42.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:43.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:44.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:44.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:44.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:39:44.289 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:39:44.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:44.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:39:44.581 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2322317129,v1:127.0.0.1:6819/2322317129] [v2:127.0.0.1:6820/2322317129,v1:127.0.0.1:6821/2322317129] exists,up c0c8408e-42ac-495a-8534-3c15d6720161 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:39:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:39:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:39:44.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 31eb0563-7c9d-4921-a411-324fd3c7a7a0' 2024-03-30T11:39:44.588 INFO:tasks.workunit.client.0.smithi003.stdout:add osd3 31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:39:44.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:39:44.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAA+gdmxYRNJBAAQgQMcQIIH7Tc9ovDDFpm6w== 2024-03-30T11:39:44.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAA+gdmxYRNJBAAQgQMcQIIH7Tc9ovDDFpm6w=="}' 2024-03-30T11:39:44.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31eb0563-7c9d-4921-a411-324fd3c7a7a0 -i td/osd-scrub-repair/3/new.json 2024-03-30T11:39:44.934 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:39:44.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-30T11:39:44.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAA+gdmxYRNJBAAQgQMcQIIH7Tc9ovDDFpm6w== --osd-uuid 31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:39:44.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:44.978+0000 7f90bcba8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:45.008 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:45.015+0000 7f90bcba8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:45.009 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:45.016+0000 7f90bcba8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:45.010 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:45.017+0000 7f90bcba8640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:39:45.010 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:45.017+0000 7f90bcba8640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:39:45.010 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:45.017+0000 7f90bcba8640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-30T11:39:45.083 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:45.090+0000 7f90bcba8640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-30T11:39:50.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-30T11:39:50.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:39:50.789 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd3 key to auth repository 2024-03-30T11:39:50.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-30T11:39:50.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:39:51.166 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.3 2024-03-30T11:39:51.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-30T11:39:51.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:39:51.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:39:51.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:39:51.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:39:51.192 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:51.199+0000 7fce900b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:51.194 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:51.201+0000 7fce900b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:51.195 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:51.202+0000 7fce900b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:39:51.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-30T11:39:51.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:39:51.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:39:51.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:39:51.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:39:51.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:51.449 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:39:51.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:39:51.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:51.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:51.713 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:51.720+0000 7fce900b5640 -1 Falling back to public interface 2024-03-30T11:39:51.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:52.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:52.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:52.731 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:39:52.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:39:52.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:52.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:53.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:54.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:54.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:54.026 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:39:54.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:39:54.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:54.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:54.100 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:39:54.107+0000 7fce900b5640 -1 osd.3 0 log_to_monitors true 2024-03-30T11:39:54.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:55.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:55.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:55.324 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:39:55.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:39:55.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:55.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:55.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:56.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:56.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:56.623 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:39:56.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:39:56.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:56.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:56.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:57.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:57.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:57.914 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:39:57.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:39:57.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:57.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:58.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:39:59.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:39:59.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:39:59.210 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:39:59.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:39:59.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:39:59.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:39:59.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:00.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:00.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:00.505 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:40:00.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:40:00.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:00.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:00.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:01.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:01.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:01.855 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:40:01.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:40:01.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:01.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:02.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:03.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:03.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:03.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:40:03.254 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:40:03.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:03.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:03.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:04.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:04.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:04.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:40:04.547 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:40:04.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:04.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:04.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:05.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:05.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:05.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:40:05.867 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:40:05.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:05.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:06.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:07.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:07.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:07.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:40:07.510 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:40:07.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:07.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:08.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:09.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:09.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:09.160 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:40:09.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:40:09.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:09.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:10.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:11.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:11.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:11.048 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:40:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:40:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:12.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:13.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:13.127 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:40:13.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:13.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:40:13.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:13.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:13.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:14.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:14.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:14.789 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:40:14.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:40:14.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:14.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:15.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:16.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:16.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:16.550 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:40:16.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:40:16.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:16.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:40:16.846 INFO:tasks.workunit.client.0.smithi003.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3647048748,v1:127.0.0.1:6827/3647048748] [v2:127.0.0.1:6828/3647048748,v1:127.0.0.1:6829/3647048748] exists,up 31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-30T11:40:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:40:16.851 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:40:16.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:40:16.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-30T11:40:16.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:40:16.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cebfd03d-8ebb-4bb1-b4ed-8598d8acdf5c 2024-03-30T11:40:16.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 cebfd03d-8ebb-4bb1-b4ed-8598d8acdf5c' 2024-03-30T11:40:16.853 INFO:tasks.workunit.client.0.smithi003.stdout:add osd4 cebfd03d-8ebb-4bb1-b4ed-8598d8acdf5c 2024-03-30T11:40:16.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:40:16.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAg+gdmDUkkNBAAGi1KXA3MYEYHS80TBDcF8w== 2024-03-30T11:40:16.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAg+gdmDUkkNBAAGi1KXA3MYEYHS80TBDcF8w=="}' 2024-03-30T11:40:16.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cebfd03d-8ebb-4bb1-b4ed-8598d8acdf5c -i td/osd-scrub-repair/4/new.json 2024-03-30T11:40:17.225 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:40:17.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-30T11:40:17.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAg+gdmDUkkNBAAGi1KXA3MYEYHS80TBDcF8w== --osd-uuid cebfd03d-8ebb-4bb1-b4ed-8598d8acdf5c 2024-03-30T11:40:17.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.272+0000 7f1ddddbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:17.318 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.325+0000 7f1ddddbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:17.318 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.326+0000 7f1ddddbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:17.319 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.326+0000 7f1ddddbd640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-30T11:40:17.319 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.326+0000 7f1ddddbd640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-30T11:40:17.319 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.326+0000 7f1ddddbd640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-30T11:40:17.418 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:17.425+0000 7f1ddddbd640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-30T11:40:22.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-30T11:40:22.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:40:22.907 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd4 key to auth repository 2024-03-30T11:40:22.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-30T11:40:22.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:40:23.275 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.4 2024-03-30T11:40:23.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-30T11:40:23.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:40:23.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:40:23.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:40:23.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:40:23.301 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:23.308+0000 7f10486f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:23.303 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:23.310+0000 7f10486f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:23.304 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:23.311+0000 7f10486f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:23.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:40:23.559 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:40:23.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:23.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:23.819 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:23.826+0000 7f10486f9640 -1 Falling back to public interface 2024-03-30T11:40:23.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:24.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:24.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:40:24.851 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:40:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:24.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:25.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:26.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:26.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:26.145 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:40:26.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:40:26.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:26.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:26.254 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:26.261+0000 7f10486f9640 -1 osd.4 0 log_to_monitors true 2024-03-30T11:40:26.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:27.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:40:27.463 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:40:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:27.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:27.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:28.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:28.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:28.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:40:28.758 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:40:28.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:28.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:29.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:30.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:30.110 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:40:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:40:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:30.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:30.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:31.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:31.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:31.405 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:40:31.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:40:31.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:31.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:31.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:32.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:32.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:32.697 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:40:32.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:40:32.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:32.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:33.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:34.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:34.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:34.077 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:40:34.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:40:34.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:34.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:34.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:35.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:35.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:35.370 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:40:35.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:40:35.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:35.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:35.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:36.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:36.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:36.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:40:36.666 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:40:36.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:36.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:36.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:37.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:37.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:37.960 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:40:37.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:40:37.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:37.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:38.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:39.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:39.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:39.479 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:40:39.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:40:39.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:39.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:41.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:41.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:41.114 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:40:41.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:40:41.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:41.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:42.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:43.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:43.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:43.113 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:40:43.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:40:43.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:43.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:43.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:44.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:44.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:44.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:40:44.980 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:40:44.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:44.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:45.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:40:46.340 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:40:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:46.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:46.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:47.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:47.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:47.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:40:47.786 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:40:47.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:47.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-30T11:40:48.076 INFO:tasks.workunit.client.0.smithi003.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2787904152,v1:127.0.0.1:6835/2787904152] [v2:127.0.0.1:6836/2787904152,v1:127.0.0.1:6837/2787904152] exists,up cebfd03d-8ebb-4bb1-b4ed-8598d8acdf5c 2024-03-30T11:40:48.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:40:48.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:40:48.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:40:48.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:40:48.085 INFO:tasks.workunit.client.0.smithi003.stdout:add osd5 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:40:48.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 9b31de6a-d097-42ce-ade3-a33a9bdc1575' 2024-03-30T11:40:48.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:40:48.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBA+gdmnjFPBhAA5a5bSlXZOrDHjYgDaaTmGg== 2024-03-30T11:40:48.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBA+gdmnjFPBhAA5a5bSlXZOrDHjYgDaaTmGg=="}' 2024-03-30T11:40:48.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b31de6a-d097-42ce-ade3-a33a9bdc1575 -i td/osd-scrub-repair/5/new.json 2024-03-30T11:40:48.503 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:40:48.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-30T11:40:48.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBA+gdmnjFPBhAA5a5bSlXZOrDHjYgDaaTmGg== --osd-uuid 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:40:48.544 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.551+0000 7fbd5348b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:48.613 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.620+0000 7fbd5348b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:48.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.621+0000 7fbd5348b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:48.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.622+0000 7fbd5348b640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-30T11:40:48.615 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.622+0000 7fbd5348b640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-30T11:40:48.615 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.622+0000 7fbd5348b640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-30T11:40:48.690 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:48.697+0000 7fbd5348b640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-30T11:40:54.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-30T11:40:54.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:40:54.391 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd5 key to auth repository 2024-03-30T11:40:54.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-30T11:40:54.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:40:54.759 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.5 2024-03-30T11:40:54.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-30T11:40:54.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:40:54.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:40:54.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:40:54.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:40:54.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:54.792+0000 7ff19200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:54.787 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:54.794+0000 7ff19200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:54.788 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:54.795+0000 7ff19200f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:40:55.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-30T11:40:55.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:40:55.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:55.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:40:55.319 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:55.326+0000 7ff19200f640 -1 Falling back to public interface 2024-03-30T11:40:55.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:56.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:56.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:56.350 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:40:56.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:40:56.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:56.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:40:56.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:57.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:57.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:57.646 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:40:57.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:40:57.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:57.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:40:57.656 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:40:57.663+0000 7ff19200f640 -1 osd.5 0 log_to_monitors true 2024-03-30T11:40:57.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:40:58.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:40:58.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:40:58.974 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:40:58.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:40:58.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:40:58.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:40:59.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:00.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:41:00.268 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:41:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:00.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:00.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:01.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:01.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:01.559 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:41:01.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:41:01.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:01.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:01.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:02.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:02.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:02.856 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:41:02.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:41:02.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:02.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:03.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:04.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:04.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:04.147 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:41:04.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:41:04.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:04.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:04.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:05.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:05.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:05.464 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:41:05.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:41:05.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:05.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:05.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:06.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:06.761 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:41:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:41:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:06.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:07.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:08.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:08.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:08.053 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:41:08.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:41:08.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:08.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:08.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:09.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:09.441 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:41:09.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:09.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:41:09.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:09.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:10.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:11.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:11.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:11.043 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:41:11.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:41:11.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:11.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:11.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:12.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:12.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:12.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:41:12.704 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:41:12.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:12.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:13.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:14.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:14.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:14.098 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:41:14.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:41:14.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:14.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:14.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:15.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:15.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:15.637 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:41:15.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:41:15.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:15.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:16.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:17.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:17.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:17.188 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:41:17.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:41:17.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:17.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:17.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:18.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:18.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:18.961 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:41:18.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:41:18.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:18.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:19.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:20.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:20.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:20.294 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:41:20.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:41:20.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:20.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:41:20.589 INFO:tasks.workunit.client.0.smithi003.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/482281156,v1:127.0.0.1:6843/482281156] [v2:127.0.0.1:6844/482281156,v1:127.0.0.1:6845/482281156] exists,up 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:41:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:41:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:41:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:41:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:41:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-30T11:41:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:41:20.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:41:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc69b926-2fd5-4f30-88ec-51bf66db5124 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stdout:add osd6 cc69b926-2fd5-4f30-88ec-51bf66db5124 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 cc69b926-2fd5-4f30-88ec-51bf66db5124' 2024-03-30T11:41:20.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:41:20.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBg+gdmgxDTJBAAYpX7OTEfHHmZlngFU6/QdQ== 2024-03-30T11:41:20.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBg+gdmgxDTJBAAYpX7OTEfHHmZlngFU6/QdQ=="}' 2024-03-30T11:41:20.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc69b926-2fd5-4f30-88ec-51bf66db5124 -i td/osd-scrub-repair/6/new.json 2024-03-30T11:41:20.971 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:41:20.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-30T11:41:20.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBg+gdmgxDTJBAAYpX7OTEfHHmZlngFU6/QdQ== --osd-uuid cc69b926-2fd5-4f30-88ec-51bf66db5124 2024-03-30T11:41:21.010 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.017+0000 7fbfdaa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:21.047 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.054+0000 7fbfdaa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:21.048 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.055+0000 7fbfdaa13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:21.048 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.055+0000 7fbfdaa13640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-30T11:41:21.048 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.055+0000 7fbfdaa13640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-30T11:41:21.049 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.055+0000 7fbfdaa13640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-30T11:41:21.114 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:21.121+0000 7fbfdaa13640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-30T11:41:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-30T11:41:27.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:41:27.130 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd6 key to auth repository 2024-03-30T11:41:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-30T11:41:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:41:27.496 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.6 2024-03-30T11:41:27.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-30T11:41:27.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:41:27.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:41:27.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:41:27.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:41:27.523 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:27.530+0000 7f02aea11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:27.524 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:27.532+0000 7f02aea11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:27.526 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:27.533+0000 7f02aea11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:27.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:41:27.861 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:41:27.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:27.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:28.041 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:28.048+0000 7f02aea11640 -1 Falling back to public interface 2024-03-30T11:41:28.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:29.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:29.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:29.145 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:41:29.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:41:29.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:29.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:29.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:30.433 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:30.440+0000 7f02aea11640 -1 osd.6 0 log_to_monitors true 2024-03-30T11:41:30.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:30.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:30.439 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:41:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:41:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:30.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:31.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:31.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:31.868 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:41:31.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:41:31.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:31.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:32.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:33.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:33.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:33.244 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:41:33.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:41:33.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:33.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:33.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:34.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:34.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:34.538 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:41:34.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:41:34.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:34.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:34.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:35.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:35.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:35.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:41:35.831 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:41:35.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:35.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:36.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:37.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:37.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:37.330 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:41:37.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:41:37.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:37.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:37.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:38.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:38.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:38.626 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:41:38.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:41:38.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:38.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:39.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:39.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:39.951 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:41:39.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:41:39.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:39.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:40.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:41.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:41.246 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:41:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:41:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:41.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:42.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:42.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:42.541 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:41:42.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:41:42.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:42.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:43.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:44.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:44.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:44.013 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:41:44.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:41:44.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:44.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:44.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:45.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:45.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:45.557 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:41:45.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:41:45.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:45.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:46.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:47.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:47.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:47.170 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:41:47.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:41:47.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:47.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:47.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:48.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:48.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:48.554 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:41:48.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:41:48.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:48.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:49.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:50.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:50.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:50.142 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:41:50.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:41:50.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:50.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:51.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:51.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:51.651 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:41:51.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:41:51.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:51.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:52.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:41:53.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:41:53.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:41:53.288 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:41:53.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:41:53.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:41:53.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-30T11:41:53.641 INFO:tasks.workunit.client.0.smithi003.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1023536900,v1:127.0.0.1:6851/1023536900] [v2:127.0.0.1:6852/1023536900,v1:127.0.0.1:6853/1023536900] exists,up cc69b926-2fd5-4f30-88ec-51bf66db5124 2024-03-30T11:41:53.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:41:53.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:41:53.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:41:53.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:41:53.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=170fefe7-8b0d-4839-a4ff-dc5890567b10 2024-03-30T11:41:53.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 170fefe7-8b0d-4839-a4ff-dc5890567b10' 2024-03-30T11:41:53.649 INFO:tasks.workunit.client.0.smithi003.stdout:add osd7 170fefe7-8b0d-4839-a4ff-dc5890567b10 2024-03-30T11:41:53.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:41:53.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCB+gdmIrXoJxAAfn2glfaw6F6l4dximsIn3A== 2024-03-30T11:41:53.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCB+gdmIrXoJxAAfn2glfaw6F6l4dximsIn3A=="}' 2024-03-30T11:41:53.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 170fefe7-8b0d-4839-a4ff-dc5890567b10 -i td/osd-scrub-repair/7/new.json 2024-03-30T11:41:54.090 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:41:54.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-30T11:41:54.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCB+gdmIrXoJxAAfn2glfaw6F6l4dximsIn3A== --osd-uuid 170fefe7-8b0d-4839-a4ff-dc5890567b10 2024-03-30T11:41:54.130 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.136+0000 7fcdd86b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:54.141 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.147+0000 7fcdd86b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:54.142 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.148+0000 7fcdd86b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:41:54.143 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.148+0000 7fcdd86b1640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-30T11:41:54.143 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.148+0000 7fcdd86b1640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-30T11:41:54.143 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.148+0000 7fcdd86b1640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-30T11:41:54.286 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:41:54.290+0000 7fcdd86b1640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-30T11:42:00.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-30T11:42:00.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:42:00.540 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd7 key to auth repository 2024-03-30T11:42:00.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-30T11:42:00.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:42:00.906 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.7 2024-03-30T11:42:00.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-30T11:42:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:42:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:42:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:42:00.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:42:00.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:00.939+0000 7fde98bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:00.934 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:00.941+0000 7fde98bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:00.935 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:00.942+0000 7fde98bff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:01.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:42:01.192 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:42:01.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:01.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:01.457 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:01.463+0000 7fde98bff640 -1 Falling back to public interface 2024-03-30T11:42:01.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:02.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:02.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:02.475 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:42:02.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:42:02.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:02.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:02.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:03.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:03.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:03.771 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:42:03.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:42:03.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:03.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:03.902+0000 7fde98bff640 -1 osd.7 0 log_to_monitors true 2024-03-30T11:42:04.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:05.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:05.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:05.156 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:42:05.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:42:05.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:05.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:05.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:06.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:06.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:06.453 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:42:06.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:42:06.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:06.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:06.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:07.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:07.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:07.753 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:42:07.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:42:07.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:07.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:08.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:09.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:09.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:09.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:42:09.100 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:42:09.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:09.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:09.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:10.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:10.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:10.395 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:42:10.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:42:10.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:10.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:10.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:11.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:11.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:11.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:42:11.745 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:42:11.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:11.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:12.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:13.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:13.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:13.102 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:42:13.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:42:13.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:13.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:13.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:14.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:14.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:14.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:42:14.394 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:42:14.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:14.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:14.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:15.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:15.773 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:42:15.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:42:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:16.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:17.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:17.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:17.622 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:42:17.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:42:17.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:17.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:18.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:19.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:19.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:19.360 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:42:19.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:42:19.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:19.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:20.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:21.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:21.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:21.383 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:42:21.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:42:21.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:21.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:22.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:23.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:23.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:23.253 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:42:23.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:42:23.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:23.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:24.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:25.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:25.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:25.098 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:42:25.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:42:25.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:25.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:25.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:26.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:26.934 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:42:26.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:26.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:42:26.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:26.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-30T11:42:27.299 INFO:tasks.workunit.client.0.smithi003.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4135176457,v1:127.0.0.1:6859/4135176457] [v2:127.0.0.1:6860/4135176457,v1:127.0.0.1:6861/4135176457] exists,up 170fefe7-8b0d-4839-a4ff-dc5890567b10 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:42:27.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:42:27.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:42:27.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b117e406-62fa-435d-acf3-04f2d36d8276 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 b117e406-62fa-435d-acf3-04f2d36d8276' 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stdout:add osd8 b117e406-62fa-435d-acf3-04f2d36d8276 2024-03-30T11:42:27.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:42:27.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCj+gdmUwaEExAAoVGu/fGsLX1X1rDaGqvyLQ== 2024-03-30T11:42:27.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCj+gdmUwaEExAAoVGu/fGsLX1X1rDaGqvyLQ=="}' 2024-03-30T11:42:27.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b117e406-62fa-435d-acf3-04f2d36d8276 -i td/osd-scrub-repair/8/new.json 2024-03-30T11:42:27.685 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:42:27.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-30T11:42:27.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCj+gdmUwaEExAAoVGu/fGsLX1X1rDaGqvyLQ== --osd-uuid b117e406-62fa-435d-acf3-04f2d36d8276 2024-03-30T11:42:27.725 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.732+0000 7f7a18823640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:27.770 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.777+0000 7f7a18823640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:27.771 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.778+0000 7f7a18823640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:27.771 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.778+0000 7f7a18823640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-30T11:42:27.771 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.778+0000 7f7a18823640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-30T11:42:27.771 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.778+0000 7f7a18823640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-30T11:42:27.845 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:27.851+0000 7f7a18823640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-30T11:42:34.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-30T11:42:34.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:42:34.471 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd8 key to auth repository 2024-03-30T11:42:34.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-30T11:42:34.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:42:34.839 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.8 2024-03-30T11:42:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-30T11:42:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:42:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:42:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:42:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:42:34.865 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:34.872+0000 7f9470e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:34.867 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:34.874+0000 7f9470e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:34.868 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:34.875+0000 7f9470e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:42:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-30T11:42:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:42:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-30T11:42:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:42:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:42:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:35.124 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:42:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:42:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:35.392 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:35.399+0000 7f9470e25640 -1 Falling back to public interface 2024-03-30T11:42:35.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:36.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:36.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:36.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:42:36.405 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:42:36.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:36.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:36.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:37.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:37.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:37.699 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:42:37.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:42:37.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:37.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:37.743 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:42:37.750+0000 7f9470e25640 -1 osd.8 0 log_to_monitors true 2024-03-30T11:42:38.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:39.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:39.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:39.012 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:42:39.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:42:39.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:39.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:39.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:40.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:40.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:40.341 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:42:40.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:42:40.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:40.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:40.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:41.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:41.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:41.634 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:42:41.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:42:41.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:41.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:42.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:43.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:43.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:43.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:42:43.048 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:42:43.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:43.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:43.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:44.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:44.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:44.388 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:42:44.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:42:44.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:44.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:44.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:45.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:45.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:45.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:42:45.754 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:42:45.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:45.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:46.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:47.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:47.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:47.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:42:47.075 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:42:47.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:47.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:47.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:48.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:48.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:48.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:42:48.397 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:42:48.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:48.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:48.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:49.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:49.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:49.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:42:49.766 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:42:49.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:49.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:50.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:51.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:51.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:51.059 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:42:51.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:42:51.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:51.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:51.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:52.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:52.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:52.372 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:42:52.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:42:52.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:52.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:52.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:53.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:53.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:53.751 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:42:53.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:42:53.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:53.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:54.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:55.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:55.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:55.503 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:42:55.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:42:55.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:55.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:56.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:57.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:57.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:57.231 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:42:57.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:42:57.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:57.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:57.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:42:58.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:42:58.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:42:58.842 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:42:58.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:42:58.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:42:58.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:42:59.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:00.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:00.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:00.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:43:00.311 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:43:00.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:00.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:43:00.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:01.737 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:43:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:43:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-30T11:43:02.070 INFO:tasks.workunit.client.0.smithi003.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3801964565,v1:127.0.0.1:6867/3801964565] [v2:127.0.0.1:6868/3801964565,v1:127.0.0.1:6869/3801964565] exists,up b117e406-62fa-435d-acf3-04f2d36d8276 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-30T11:43:02.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:43:02.075 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:43:02.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:43:02.077 INFO:tasks.workunit.client.0.smithi003.stdout:add osd9 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:43:02.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:43:02.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 53b6c8e8-14ff-41f7-b0be-ac0444cedf48' 2024-03-30T11:43:02.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:43:02.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDG+gdmPI3vBRAAMjtAWCJHJ6m+oQQCCAYesw== 2024-03-30T11:43:02.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDG+gdmPI3vBRAAMjtAWCJHJ6m+oQQCCAYesw=="}' 2024-03-30T11:43:02.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 -i td/osd-scrub-repair/9/new.json 2024-03-30T11:43:02.436 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:43:02.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-30T11:43:02.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDG+gdmPI3vBRAAMjtAWCJHJ6m+oQQCCAYesw== --osd-uuid 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:43:02.476 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.482+0000 7ff026cf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:43:02.512 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.518+0000 7ff026cf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:43:02.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.519+0000 7ff026cf5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:43:02.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.519+0000 7ff026cf5640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-30T11:43:02.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.519+0000 7ff026cf5640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-30T11:43:02.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.519+0000 7ff026cf5640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-30T11:43:02.586 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:02.592+0000 7ff026cf5640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-30T11:43:08.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-30T11:43:08.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:43:08.647 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd9 key to auth repository 2024-03-30T11:43:08.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-30T11:43:08.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:43:09.013 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.9 2024-03-30T11:43:09.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-30T11:43:09.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:43:09.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:43:09.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:43:09.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:43:09.039 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:09.046+0000 7f1b15e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:43:09.055 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:09.062+0000 7f1b15e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:43:09.056 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:09.063+0000 7f1b15e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:09.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:43:09.332 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:43:09.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:09.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:09.592 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:09.599+0000 7f1b15e0f640 -1 Falling back to public interface 2024-03-30T11:43:09.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:10.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:10.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:10.612 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:43:10.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:43:10.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:10.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:10.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:11.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:11.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:11.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:43:11.920 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:43:11.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:11.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:12.024 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:43:12.031+0000 7f1b15e0f640 -1 osd.9 0 log_to_monitors true 2024-03-30T11:43:12.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:13.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:13.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:13.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:43:13.241 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:43:13.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:13.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:13.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:14.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:14.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:14.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:43:14.537 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:43:14.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:14.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:14.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:15.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:15.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:15.894 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:43:15.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:43:15.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:15.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:16.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:17.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:17.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:17.431 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:43:17.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:43:17.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:17.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:17.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:18.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:18.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:18.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:43:18.727 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:43:18.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:18.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:19.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:20.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:20.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:20.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:43:20.053 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:43:20.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:20.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:20.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:21.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:21.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:21.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:43:21.447 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:43:21.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:21.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:21.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:22.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:22.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:22.741 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:43:22.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:43:22.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:22.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:23.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:24.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:24.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:24.064 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:43:24.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:43:24.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:24.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:24.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:25.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:25.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:25.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:43:25.363 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:43:25.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:25.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:26.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:27.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:27.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:27.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:43:27.262 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:43:27.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:27.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:28.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:29.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:29.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:29.006 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:43:29.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:43:29.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:29.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:29.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:30.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:30.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:30.591 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:43:30.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:43:30.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:30.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:31.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:32.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:32.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:32.120 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:43:32.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:43:32.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:32.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:32.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:33.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:33.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:33.766 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:43:33.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:43:33.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:33.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:34.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:43:35.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:43:35.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:43:35.369 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:43:35.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:43:35.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:43:35.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:43:35.717 INFO:tasks.workunit.client.0.smithi003.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2489790527,v1:127.0.0.1:6875/2489790527] [v2:127.0.0.1:6876/2489790527,v1:127.0.0.1:6877/2489790527] exists,up 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:43:35.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:43:35.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:43:35.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:43:35.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-30T11:43:35.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:43:35.995 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:43:36.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:43:36.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:43:36.380 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:43:36.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:43:37.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:43:40.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-30T11:43:40.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:43:40.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:43:40.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:43:40.416 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:43:40.417 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:43:40.417 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:43:40.417 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:43:40.417 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:43:40.417 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:43:40.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:43:40.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:43:40.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:43:40.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:43:40.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:43:40.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:43:40.802 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:43:40.803 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:43:40.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:43:40.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:40.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:43:40.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-03-30T11:43:40.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-03-30T11:43:40.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-03-30T11:43:40.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:40.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:43:41.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673017 2024-03-30T11:43:41.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673017 2024-03-30T11:43:41.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017' 2024-03-30T11:43:41.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:43:41.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509492 2024-03-30T11:43:41.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509492 2024-03-30T11:43:41.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492' 2024-03-30T11:43:41.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.117 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:43:41.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345968 2024-03-30T11:43:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345968 2024-03-30T11:43:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968' 2024-03-30T11:43:41.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:43:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182440 2024-03-30T11:43:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182440 2024-03-30T11:43:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968 4-107374182440' 2024-03-30T11:43:41.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:43:41.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018911 2024-03-30T11:43:41.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018911 2024-03-30T11:43:41.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968 4-107374182440 5-128849018911' 2024-03-30T11:43:41.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:43:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855388 2024-03-30T11:43:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855388 2024-03-30T11:43:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968 4-107374182440 5-128849018911 6-150323855388' 2024-03-30T11:43:41.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.539 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:43:41.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691857 2024-03-30T11:43:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691857 2024-03-30T11:43:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968 4-107374182440 5-128849018911 6-150323855388 7-171798691857' 2024-03-30T11:43:41.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:43:41.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528332 2024-03-30T11:43:41.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528332 2024-03-30T11:43:41.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968 4-107374182440 5-128849018911 6-150323855388 7-171798691857 8-193273528332' 2024-03-30T11:43:41.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:41.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:43:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-03-30T11:43:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-03-30T11:43:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-42949673017 2-64424509492 3-85899345968 4-107374182440 5-128849018911 6-150323855388 7-171798691857 8-193273528332 9-214748364806' 2024-03-30T11:43:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:41.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-03-30T11:43:41.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:43:41.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-03-30T11:43:41.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-03-30T11:43:41.858 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836544 2024-03-30T11:43:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-03-30T11:43:41.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:43:42.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836543 -lt 21474836544 2024-03-30T11:43:42.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:43:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:43:43.140 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:43:43.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-03-30T11:43:43.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:43.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673017 2024-03-30T11:43:43.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:43.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:43:43.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673017 2024-03-30T11:43:43.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:43.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673017 2024-03-30T11:43:43.437 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949673017 2024-03-30T11:43:43.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2024-03-30T11:43:43.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:43:43.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673018 -lt 42949673017 2024-03-30T11:43:43.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:43.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509492 2024-03-30T11:43:43.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:43.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:43:43.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509492 2024-03-30T11:43:43.722 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:43.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509492 2024-03-30T11:43:43.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2024-03-30T11:43:43.725 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509492 2024-03-30T11:43:43.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:43:44.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509492 -lt 64424509492 2024-03-30T11:43:44.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:44.009 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345968 2024-03-30T11:43:44.009 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:44.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:43:44.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345968 2024-03-30T11:43:44.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:44.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345968 2024-03-30T11:43:44.012 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345968 2024-03-30T11:43:44.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2024-03-30T11:43:44.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:43:44.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345969 -lt 85899345968 2024-03-30T11:43:44.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:44.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182440 2024-03-30T11:43:44.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:44.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:43:44.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182440 2024-03-30T11:43:44.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:44.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182440 2024-03-30T11:43:44.300 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.4 seq 107374182440 2024-03-30T11:43:44.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2024-03-30T11:43:44.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:43:44.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182441 -lt 107374182440 2024-03-30T11:43:44.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:44.583 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018911 2024-03-30T11:43:44.583 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:44.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:43:44.585 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018911 2024-03-30T11:43:44.585 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:44.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018911 2024-03-30T11:43:44.585 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.5 seq 128849018911 2024-03-30T11:43:44.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018911' 2024-03-30T11:43:44.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:43:44.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018911 -lt 128849018911 2024-03-30T11:43:44.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:44.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855388 2024-03-30T11:43:44.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:44.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:43:44.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855388 2024-03-30T11:43:44.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:44.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855388 2024-03-30T11:43:44.872 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.6 seq 150323855388 2024-03-30T11:43:44.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-03-30T11:43:44.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:43:45.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855389 -lt 150323855388 2024-03-30T11:43:45.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:45.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691857 2024-03-30T11:43:45.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:45.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:43:45.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691857 2024-03-30T11:43:45.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:45.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691857 2024-03-30T11:43:45.173 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.7 seq 171798691857 2024-03-30T11:43:45.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691857' 2024-03-30T11:43:45.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:43:45.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691858 -lt 171798691857 2024-03-30T11:43:45.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:45.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528332 2024-03-30T11:43:45.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:45.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:43:45.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528332 2024-03-30T11:43:45.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:45.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528332 2024-03-30T11:43:45.458 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.8 seq 193273528332 2024-03-30T11:43:45.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528332' 2024-03-30T11:43:45.458 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:43:45.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528333 -lt 193273528332 2024-03-30T11:43:45.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:45.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364806 2024-03-30T11:43:45.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:45.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:43:45.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364806 2024-03-30T11:43:45.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:45.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-03-30T11:43:45.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364806' 2024-03-30T11:43:45.742 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.9 seq 214748364806 2024-03-30T11:43:45.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:43:46.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-03-30T11:43:46.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:43:46.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:43:46.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:43:46.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:43:46.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:43:46.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:43:46.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:43:46.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:43:46.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:43:46.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:43:46.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:43:46.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:43:46.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:43:46.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:43:46.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:43:47.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:43:47.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:43:47.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:43:47.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-03-30T11:43:47.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:43:47.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:43:47.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-30T11:43:47.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:43:47.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-30T11:43:47.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:43:47.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:43:48.611 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:43:48.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:43:49.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-30T11:43:49.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-30T11:43:50.106 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 2 allow_ec_overwrites to true 2024-03-30T11:43:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:43:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:43:50.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:43:50.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:43:50.118 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:43:50.119 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:43:50.119 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:43:50.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:43:50.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:43:50.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:43:50.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:43:50.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:43:50.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:43:50.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:43:50.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:43:50.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:43:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:43:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:43:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:43:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:43:50.648 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:50.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:43:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-03-30T11:43:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-03-30T11:43:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-03-30T11:43:50.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:50.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:43:51.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-03-30T11:43:51.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-03-30T11:43:51.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020' 2024-03-30T11:43:51.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:43:51.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509495 2024-03-30T11:43:51.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509495 2024-03-30T11:43:51.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495' 2024-03-30T11:43:51.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.242 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:43:51.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345971 2024-03-30T11:43:51.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345971 2024-03-30T11:43:51.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971' 2024-03-30T11:43:51.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:43:51.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182443 2024-03-30T11:43:51.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182443 2024-03-30T11:43:51.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971 4-107374182443' 2024-03-30T11:43:51.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:43:51.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-03-30T11:43:51.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-03-30T11:43:51.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971 4-107374182443 5-128849018914' 2024-03-30T11:43:51.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:43:51.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855391 2024-03-30T11:43:51.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855391 2024-03-30T11:43:51.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971 4-107374182443 5-128849018914 6-150323855391' 2024-03-30T11:43:51.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:43:51.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691860 2024-03-30T11:43:51.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691860 2024-03-30T11:43:51.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971 4-107374182443 5-128849018914 6-150323855391 7-171798691860' 2024-03-30T11:43:51.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:43:51.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528335 2024-03-30T11:43:51.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528335 2024-03-30T11:43:51.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971 4-107374182443 5-128849018914 6-150323855391 7-171798691860 8-193273528335' 2024-03-30T11:43:51.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:43:51.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:43:52.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364809 2024-03-30T11:43:52.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364809 2024-03-30T11:43:52.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673020 2-64424509495 3-85899345971 4-107374182443 5-128849018914 6-150323855391 7-171798691860 8-193273528335 9-214748364809' 2024-03-30T11:43:52.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:52.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-03-30T11:43:52.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:52.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:43:52.070 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-03-30T11:43:52.070 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:52.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-03-30T11:43:52.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-03-30T11:43:52.072 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836547 2024-03-30T11:43:52.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:43:52.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836547 2024-03-30T11:43:52.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:52.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-03-30T11:43:52.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:52.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:43:52.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-03-30T11:43:52.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:52.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-03-30T11:43:52.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-03-30T11:43:52.357 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949673020 2024-03-30T11:43:52.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:43:52.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673019 -lt 42949673020 2024-03-30T11:43:52.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:43:53.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:43:53.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:43:53.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673020 2024-03-30T11:43:53.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:53.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509495 2024-03-30T11:43:53.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:53.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:43:53.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509495 2024-03-30T11:43:53.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:53.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509495 2024-03-30T11:43:53.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2024-03-30T11:43:53.941 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509495 2024-03-30T11:43:53.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:43:54.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509495 -lt 64424509495 2024-03-30T11:43:54.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:54.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345971 2024-03-30T11:43:54.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:54.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:43:54.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345971 2024-03-30T11:43:54.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:54.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345971 2024-03-30T11:43:54.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345971' 2024-03-30T11:43:54.229 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.3 seq 85899345971 2024-03-30T11:43:54.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:43:54.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345972 -lt 85899345971 2024-03-30T11:43:54.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:54.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182443 2024-03-30T11:43:54.515 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:54.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:43:54.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182443 2024-03-30T11:43:54.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:54.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182443 2024-03-30T11:43:54.517 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.4 seq 107374182443 2024-03-30T11:43:54.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2024-03-30T11:43:54.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:43:54.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182444 -lt 107374182443 2024-03-30T11:43:54.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:54.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018914 2024-03-30T11:43:54.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:54.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:43:54.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018914 2024-03-30T11:43:54.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:54.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-03-30T11:43:54.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018914' 2024-03-30T11:43:54.802 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.5 seq 128849018914 2024-03-30T11:43:54.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:43:55.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018914 -lt 128849018914 2024-03-30T11:43:55.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:55.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855391 2024-03-30T11:43:55.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:55.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:43:55.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855391 2024-03-30T11:43:55.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:55.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855391 2024-03-30T11:43:55.088 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.6 seq 150323855391 2024-03-30T11:43:55.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2024-03-30T11:43:55.088 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:43:55.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855392 -lt 150323855391 2024-03-30T11:43:55.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:55.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691860 2024-03-30T11:43:55.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:55.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:43:55.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691860 2024-03-30T11:43:55.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:55.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691860 2024-03-30T11:43:55.412 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.7 seq 171798691860 2024-03-30T11:43:55.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691860' 2024-03-30T11:43:55.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:43:55.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691861 -lt 171798691860 2024-03-30T11:43:55.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:55.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528335 2024-03-30T11:43:55.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:55.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:43:55.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528335 2024-03-30T11:43:55.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:55.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528335 2024-03-30T11:43:55.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-03-30T11:43:55.701 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.8 seq 193273528335 2024-03-30T11:43:55.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:43:55.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528335 2024-03-30T11:43:55.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:43:55.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364809 2024-03-30T11:43:55.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:43:55.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:43:55.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364809 2024-03-30T11:43:55.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:43:55.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364809 2024-03-30T11:43:55.986 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.9 seq 214748364809 2024-03-30T11:43:55.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-03-30T11:43:55.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:43:56.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364809 -lt 214748364809 2024-03-30T11:43:56.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:43:56.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:43:56.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:43:56.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:43:56.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:43:56.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:43:56.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:43:56.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:43:56.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:43:56.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:43:56.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:43:56.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:43:56.905 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:43:56.905 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:43:56.905 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:43:57.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:43:57.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:43:57.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:43:57.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:43:57.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:43:57.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:43:57.646 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:43:57.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:43:58.224 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:43:58.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:43:58.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:43:58.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:43:58.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-30T11:43:58.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-30T11:43:58.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:43:58.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:43:58.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:43:58.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-30T11:43:58.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-30T11:43:58.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T11:43:58.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-30T11:43:58.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-30T11:43:58.964 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:43:58.964 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T11:43:59.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:9 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr:7' 2024-03-30T11:43:59.254 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:43:59.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:43:59.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:43:59.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:43:59.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:43:59.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:43:59.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:44:01.313 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:44:01.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:44:01.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:44:01.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:44:01.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:44:01.831 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:44:01.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:44:01.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:44:01.833 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.3 2024-03-30T11:44:01.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:44:01.834 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:44:01.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:44:01.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:44:01.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:44:01.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:44:01.860 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:01.867+0000 7f5f5fa68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:01.874 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:01.881+0000 7f5f5fa68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:01.875 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:01.882+0000 7f5f5fa68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:02.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:44:02.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:44:02.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:02.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:44:02.395 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:02.402+0000 7f5f5fa68640 -1 Falling back to public interface 2024-03-30T11:44:02.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:44:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:03.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:03.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:44:03.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:04.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:04.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:04.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:44:04.701 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:04.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:04.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:44:04.728 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:04.735+0000 7f5f5fa68640 -1 osd.3 67 log_to_monitors true 2024-03-30T11:44:05.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:06.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:06.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:06.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:44:06.068 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:06.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:06.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:44:06.403 INFO:tasks.workunit.client.0.smithi003.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/1365943850,v1:127.0.0.1:6827/1365943850] [v2:127.0.0.1:6828/1365943850,v1:127.0.0.1:6829/1365943850] exists,up 31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:44:06.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:44:06.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:44:06.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:44:06.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:44:06.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:44:06.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:44:06.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:44:06.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:44:06.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:44:06.831 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:06.831 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:06.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:44:06.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836551 2024-03-30T11:44:06.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836551 2024-03-30T11:44:06.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551' 2024-03-30T11:44:06.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:06.979 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:44:07.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-03-30T11:44:07.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-03-30T11:44:07.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025' 2024-03-30T11:44:07.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:44:07.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509499 2024-03-30T11:44:07.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509499 2024-03-30T11:44:07.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499' 2024-03-30T11:44:07.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:44:07.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-03-30T11:44:07.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-03-30T11:44:07.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018' 2024-03-30T11:44:07.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:44:07.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182447 2024-03-30T11:44:07.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182447 2024-03-30T11:44:07.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018 4-107374182447' 2024-03-30T11:44:07.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.506 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:44:07.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-03-30T11:44:07.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-03-30T11:44:07.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018 4-107374182447 5-128849018918' 2024-03-30T11:44:07.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:44:07.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855395 2024-03-30T11:44:07.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855395 2024-03-30T11:44:07.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018 4-107374182447 5-128849018918 6-150323855395' 2024-03-30T11:44:07.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:44:07.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691865 2024-03-30T11:44:07.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691865 2024-03-30T11:44:07.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018 4-107374182447 5-128849018918 6-150323855395 7-171798691865' 2024-03-30T11:44:07.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:44:07.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-03-30T11:44:07.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-03-30T11:44:07.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018 4-107374182447 5-128849018918 6-150323855395 7-171798691865 8-193273528340' 2024-03-30T11:44:07.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:07.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:44:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364813 2024-03-30T11:44:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364813 2024-03-30T11:44:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836551 1-42949673025 2-64424509499 3-304942678018 4-107374182447 5-128849018918 6-150323855395 7-171798691865 8-193273528340 9-214748364813' 2024-03-30T11:44:08.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:08.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836551 2024-03-30T11:44:08.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:08.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:44:08.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836551 2024-03-30T11:44:08.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:08.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836551 2024-03-30T11:44:08.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836551' 2024-03-30T11:44:08.055 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836551 2024-03-30T11:44:08.055 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:44:08.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836551 2024-03-30T11:44:08.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:44:09.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:44:09.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:44:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836551 2024-03-30T11:44:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:09.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-03-30T11:44:09.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:44:09.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-03-30T11:44:09.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:09.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-03-30T11:44:09.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-03-30T11:44:09.634 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673025 2024-03-30T11:44:09.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:44:09.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-03-30T11:44:09.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:09.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509499 2024-03-30T11:44:09.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:09.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:44:09.932 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509499 2024-03-30T11:44:09.932 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:09.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509499 2024-03-30T11:44:09.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-03-30T11:44:09.933 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509499 2024-03-30T11:44:09.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:44:10.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509499 -lt 64424509499 2024-03-30T11:44:10.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:10.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-03-30T11:44:10.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:10.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:44:10.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-03-30T11:44:10.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:10.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-03-30T11:44:10.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-03-30T11:44:10.221 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678018 2024-03-30T11:44:10.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:44:10.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-03-30T11:44:10.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:10.504 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182447 2024-03-30T11:44:10.504 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:10.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:44:10.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182447 2024-03-30T11:44:10.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:10.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182447 2024-03-30T11:44:10.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2024-03-30T11:44:10.506 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182447 2024-03-30T11:44:10.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:44:10.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182447 2024-03-30T11:44:10.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:10.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018918 2024-03-30T11:44:10.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:10.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:44:10.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018918 2024-03-30T11:44:10.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-03-30T11:44:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-03-30T11:44:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 128849018918 2024-03-30T11:44:10.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:44:11.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018919 -lt 128849018918 2024-03-30T11:44:11.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:11.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855395 2024-03-30T11:44:11.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:11.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:44:11.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855395 2024-03-30T11:44:11.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:11.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855395 2024-03-30T11:44:11.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-03-30T11:44:11.076 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855395 2024-03-30T11:44:11.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:44:11.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855396 -lt 150323855395 2024-03-30T11:44:11.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:11.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691865 2024-03-30T11:44:11.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:11.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:44:11.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691865 2024-03-30T11:44:11.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:11.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691865 2024-03-30T11:44:11.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-03-30T11:44:11.375 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691865 2024-03-30T11:44:11.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:44:11.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691865 2024-03-30T11:44:11.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:11.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-03-30T11:44:11.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:11.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:44:11.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-03-30T11:44:11.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:11.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-03-30T11:44:11.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-03-30T11:44:11.659 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528340 2024-03-30T11:44:11.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:44:11.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528340 2024-03-30T11:44:11.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:11.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364813 2024-03-30T11:44:11.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:11.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:44:11.954 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364813 2024-03-30T11:44:11.954 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:11.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364813 2024-03-30T11:44:11.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-03-30T11:44:11.955 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 214748364813 2024-03-30T11:44:11.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:44:12.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364814 -lt 214748364813 2024-03-30T11:44:12.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:44:12.237 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:44:12.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:44:12.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:44:12.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:44:12.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:44:12.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:44:12.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:44:12.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:44:12.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:44:12.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:44:12.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:44:12.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:44:12.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:44:12.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:44:13.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:44:13.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:44:13.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:44:13.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:44:13.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:44:13.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:44:13.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:44:13.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:44:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:44:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:44:13.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:44:13.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:44:13.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:13.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:13.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:13.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:13.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:43:48.377284+0000 2024-03-30T11:44:13.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:44:14.076 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:44:14.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:43:48.377284+0000 2024-03-30T11:44:14.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:44:14.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:43:48.377284+0000 2024-03-30T11:44:14.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:14.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:14.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:43:48.377284+0000 '>' 2024-03-30T11:43:48.377284+0000 2024-03-30T11:44:14.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:44:15.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:44:15.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:15.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:15.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:15.367 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:15.367 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:15.367 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:15.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:43:48.377284+0000 '>' 2024-03-30T11:43:48.377284+0000 2024-03-30T11:44:15.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:44:16.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:44:16.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:16.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:16.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:16.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:16.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:16.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:16.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:14.707264+0000 '>' 2024-03-30T11:43:48.377284+0000 2024-03-30T11:44:16.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:44:16.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:44:16.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:44:16.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:44:17.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:44:17.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:44:17.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:44:17.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:44:17.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:44:17.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:44:17.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:44:17.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-30T11:44:19.790 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:44:19.790 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:44:19.790 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:44:20.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:20.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:44:20.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:44:20.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:44:20.063 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.9 2024-03-30T11:44:20.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:44:20.063 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:44:20.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:44:20.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:44:20.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:44:20.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:44:20.090 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:20.097+0000 7fa8c480c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:20.093 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:20.100+0000 7fa8c480c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:20.094 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:20.101+0000 7fa8c480c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:20.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:44:20.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:20.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:20.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:20.621+0000 7fa8c480c640 -1 Falling back to public interface 2024-03-30T11:44:20.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:21.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:21.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:44:21.644 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:21.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:21.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:21.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:22.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:22.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:22.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:44:22.940 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:22.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:22.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:22.957 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:22.954+0000 7fa8c480c640 -1 osd.9 72 log_to_monitors true 2024-03-30T11:44:23.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:24.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:24.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:24.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:44:24.269 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:24.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:24.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:24.748 INFO:tasks.workunit.client.0.smithi003.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/1453025918,v1:127.0.0.1:6875/1453025918] [v2:127.0.0.1:6876/1453025918,v1:127.0.0.1:6877/1453025918] exists,up 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:44:24.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:44:24.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:44:24.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:44:24.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:44:24.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:44:24.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:44:24.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:44:24.833 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:44:25.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:44:25.405 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:25.405 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:25.405 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:25.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:44:25.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-03-30T11:44:25.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-03-30T11:44:25.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-03-30T11:44:25.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:25.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:44:25.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673029 2024-03-30T11:44:25.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673029 2024-03-30T11:44:25.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029' 2024-03-30T11:44:25.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:25.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:44:25.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509504 2024-03-30T11:44:25.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509504 2024-03-30T11:44:25.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504' 2024-03-30T11:44:25.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:25.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:44:25.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678023 2024-03-30T11:44:25.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678023 2024-03-30T11:44:25.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023' 2024-03-30T11:44:25.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:25.908 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:44:26.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-03-30T11:44:26.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-03-30T11:44:26.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023 4-107374182452' 2024-03-30T11:44:26.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:26.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:44:26.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-03-30T11:44:26.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-03-30T11:44:26.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023 4-107374182452 5-128849018923' 2024-03-30T11:44:26.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:26.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:44:26.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855400 2024-03-30T11:44:26.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855400 2024-03-30T11:44:26.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023 4-107374182452 5-128849018923 6-150323855400' 2024-03-30T11:44:26.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:26.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:44:26.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691869 2024-03-30T11:44:26.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691869 2024-03-30T11:44:26.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023 4-107374182452 5-128849018923 6-150323855400 7-171798691869' 2024-03-30T11:44:26.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:26.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:44:26.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528344 2024-03-30T11:44:26.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528344 2024-03-30T11:44:26.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023 4-107374182452 5-128849018923 6-150323855400 7-171798691869 8-193273528344' 2024-03-30T11:44:26.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:26.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:44:26.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:44:26.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:44:26.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509504 3-304942678023 4-107374182452 5-128849018923 6-150323855400 7-171798691869 8-193273528344 9-330712481794' 2024-03-30T11:44:26.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:26.575 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-03-30T11:44:26.575 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:26.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:44:26.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-03-30T11:44:26.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:26.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-03-30T11:44:26.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-03-30T11:44:26.578 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836556 2024-03-30T11:44:26.578 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:44:26.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836556 2024-03-30T11:44:26.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:44:27.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:44:27.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:44:28.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-03-30T11:44:28.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:28.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673029 2024-03-30T11:44:28.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:28.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:44:28.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673029 2024-03-30T11:44:28.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:28.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673029 2024-03-30T11:44:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-03-30T11:44:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673029 2024-03-30T11:44:28.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:44:28.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673029 -lt 42949673029 2024-03-30T11:44:28.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:28.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509504 2024-03-30T11:44:28.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:28.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:44:28.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509504 2024-03-30T11:44:28.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:28.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509504 2024-03-30T11:44:28.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509504' 2024-03-30T11:44:28.444 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509504 2024-03-30T11:44:28.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:44:28.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509504 -lt 64424509504 2024-03-30T11:44:28.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:28.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678023 2024-03-30T11:44:28.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:28.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:44:28.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678023 2024-03-30T11:44:28.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:28.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678023 2024-03-30T11:44:28.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-03-30T11:44:28.737 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678023 2024-03-30T11:44:28.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:44:29.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678023 2024-03-30T11:44:29.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:29.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-03-30T11:44:29.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:29.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:44:29.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-03-30T11:44:29.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:29.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-03-30T11:44:29.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-03-30T11:44:29.024 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182452 2024-03-30T11:44:29.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:44:29.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182453 -lt 107374182452 2024-03-30T11:44:29.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:29.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018923 2024-03-30T11:44:29.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:29.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:44:29.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018923 2024-03-30T11:44:29.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:29.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-03-30T11:44:29.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-03-30T11:44:29.312 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 128849018923 2024-03-30T11:44:29.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:44:29.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018923 -lt 128849018923 2024-03-30T11:44:29.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:29.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855400 2024-03-30T11:44:29.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:29.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:44:29.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855400 2024-03-30T11:44:29.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:29.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855400 2024-03-30T11:44:29.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-03-30T11:44:29.595 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855400 2024-03-30T11:44:29.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:44:29.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855400 -lt 150323855400 2024-03-30T11:44:29.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:29.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691869 2024-03-30T11:44:29.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:29.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:44:29.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691869 2024-03-30T11:44:29.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:29.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691869 2024-03-30T11:44:29.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2024-03-30T11:44:29.892 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691869 2024-03-30T11:44:29.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:44:30.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691870 -lt 171798691869 2024-03-30T11:44:30.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:30.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528344 2024-03-30T11:44:30.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:30.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:44:30.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528344 2024-03-30T11:44:30.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:30.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528344 2024-03-30T11:44:30.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-03-30T11:44:30.185 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528344 2024-03-30T11:44:30.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:44:30.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528345 -lt 193273528344 2024-03-30T11:44:30.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:30.468 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-03-30T11:44:30.468 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:30.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:44:30.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-03-30T11:44:30.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:30.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:44:30.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-03-30T11:44:30.471 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 330712481794 2024-03-30T11:44:30.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:44:30.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-03-30T11:44:30.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:44:30.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:44:30.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:44:31.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:44:31.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:44:31.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:44:31.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:44:31.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:44:31.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:44:31.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:44:31.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:44:31.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:44:31.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:44:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-30T11:44:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:44:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-30T11:44:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-30T11:44:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:44:31.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:44:32.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:44:32.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:44:32.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:44:32.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:44:32.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:44:32.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:44:32.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:44:32.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-30T11:44:34.032 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:44:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:44:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:44:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:44:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.5 2024-03-30T11:44:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:44:34.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:44:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:44:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:44:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:44:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:44:34.580 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:34.587+0000 7f570900d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:34.583 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:34.590+0000 7f570900d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:34.584 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:34.591+0000 7f570900d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:44:34.848 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:44:34.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:34.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:44:35.101 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:35.108+0000 7f570900d640 -1 Falling back to public interface 2024-03-30T11:44:35.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:36.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:36.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:36.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:44:36.132 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:36.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:36.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:44:36.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:37.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:37.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:37.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:44:37.425 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:37.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:37.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:44:37.483 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:37.490+0000 7f570900d640 -1 osd.5 79 log_to_monitors true 2024-03-30T11:44:37.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:44:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:38.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:44:39.176 INFO:tasks.workunit.client.0.smithi003.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/110827939,v1:127.0.0.1:6843/110827939] [v2:127.0.0.1:6844/110827939,v1:127.0.0.1:6845/110827939] exists,up 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:44:39.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:44:39.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:44:39.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:44:39.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:44:39.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:44:39.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:44:39.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:44:39.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:44:39.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:44:39.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:44:39.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:44:39.261 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:44:39.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:39.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:44:39.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836560 2024-03-30T11:44:39.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836560 2024-03-30T11:44:39.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560' 2024-03-30T11:44:39.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:39.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:44:40.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-03-30T11:44:40.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-03-30T11:44:40.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033' 2024-03-30T11:44:40.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:44:40.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509508 2024-03-30T11:44:40.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509508 2024-03-30T11:44:40.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508' 2024-03-30T11:44:40.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:44:40.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678027 2024-03-30T11:44:40.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678027 2024-03-30T11:44:40.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027' 2024-03-30T11:44:40.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.240 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:44:40.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182456 2024-03-30T11:44:40.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182456 2024-03-30T11:44:40.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027 4-107374182456' 2024-03-30T11:44:40.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:44:40.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-03-30T11:44:40.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-03-30T11:44:40.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027 4-107374182456 5-356482285570' 2024-03-30T11:44:40.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:44:40.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855404 2024-03-30T11:44:40.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855404 2024-03-30T11:44:40.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027 4-107374182456 5-356482285570 6-150323855404' 2024-03-30T11:44:40.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:44:40.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-03-30T11:44:40.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-03-30T11:44:40.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027 4-107374182456 5-356482285570 6-150323855404 7-171798691873' 2024-03-30T11:44:40.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:44:40.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528348 2024-03-30T11:44:40.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528348 2024-03-30T11:44:40.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027 4-107374182456 5-356482285570 6-150323855404 7-171798691873 8-193273528348' 2024-03-30T11:44:40.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:40.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:44:40.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-03-30T11:44:40.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-03-30T11:44:40.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509508 3-304942678027 4-107374182456 5-356482285570 6-150323855404 7-171798691873 8-193273528348 9-330712481798' 2024-03-30T11:44:40.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:40.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836560 2024-03-30T11:44:40.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:40.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:44:40.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836560 2024-03-30T11:44:40.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:40.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836560 2024-03-30T11:44:40.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-03-30T11:44:40.874 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836560 2024-03-30T11:44:40.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:44:41.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836560 2024-03-30T11:44:41.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:41.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-03-30T11:44:41.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:41.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:44:41.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-03-30T11:44:41.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-03-30T11:44:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-03-30T11:44:41.163 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673033 2024-03-30T11:44:41.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:44:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673033 -lt 42949673033 2024-03-30T11:44:41.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:41.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509508 2024-03-30T11:44:41.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:41.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:44:41.448 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509508 2024-03-30T11:44:41.448 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509508 2024-03-30T11:44:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-03-30T11:44:41.449 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509508 2024-03-30T11:44:41.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:44:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509508 2024-03-30T11:44:41.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:41.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678027 2024-03-30T11:44:41.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:41.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:44:41.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678027 2024-03-30T11:44:41.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678027 2024-03-30T11:44:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678027' 2024-03-30T11:44:41.734 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678027 2024-03-30T11:44:41.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:44:42.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678027 -lt 304942678027 2024-03-30T11:44:42.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:42.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182456 2024-03-30T11:44:42.021 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:42.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:44:42.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182456 2024-03-30T11:44:42.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:42.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182456 2024-03-30T11:44:42.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-03-30T11:44:42.024 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182456 2024-03-30T11:44:42.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:44:42.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182456 -lt 107374182456 2024-03-30T11:44:42.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:42.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285570 2024-03-30T11:44:42.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:42.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:44:42.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285570 2024-03-30T11:44:42.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:42.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-03-30T11:44:42.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-03-30T11:44:42.310 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 356482285570 2024-03-30T11:44:42.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:44:42.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285570 -lt 356482285570 2024-03-30T11:44:42.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:42.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855404 2024-03-30T11:44:42.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:42.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:44:42.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855404 2024-03-30T11:44:42.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:42.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855404 2024-03-30T11:44:42.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855404' 2024-03-30T11:44:42.595 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855404 2024-03-30T11:44:42.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:44:42.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855404 -lt 150323855404 2024-03-30T11:44:42.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:42.878 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-03-30T11:44:42.878 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:42.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:44:42.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-03-30T11:44:42.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:42.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-03-30T11:44:42.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-03-30T11:44:42.881 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691873 2024-03-30T11:44:42.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:44:43.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691873 2024-03-30T11:44:43.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:43.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528348 2024-03-30T11:44:43.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:43.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:44:43.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528348 2024-03-30T11:44:43.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:43.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528348 2024-03-30T11:44:43.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-03-30T11:44:43.183 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528348 2024-03-30T11:44:43.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:44:43.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528349 -lt 193273528348 2024-03-30T11:44:43.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:43.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-03-30T11:44:43.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:43.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:44:43.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-03-30T11:44:43.468 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:43.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-03-30T11:44:43.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-03-30T11:44:43.469 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 330712481798 2024-03-30T11:44:43.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:44:43.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-03-30T11:44:43.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:44:43.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:44:43.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:44:44.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:44:44.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:44:44.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:44:44.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:44:44.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:44:44.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:45.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:45.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:45.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:45.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:44:45.580 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:45.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:45.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:45.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:45.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:14.707264+0000 '>' 2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:45.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:44:46.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:44:46.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:46.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:46.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:46.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:46.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:46.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:47.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:14.707264+0000 '>' 2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:47.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:44:48.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:44:48.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:48.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:48.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:48.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:48.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:48.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:48.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:14.707264+0000 '>' 2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:48.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:44:49.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:44:49.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:49.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:49.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:49.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:49.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:49.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:49.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:14.707264+0000 '>' 2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:49.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:44:50.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:44:50.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:44:50.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:44:50.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:44:50.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:44:50.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:44:50.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:45.752801+0000 '>' 2024-03-30T11:44:14.707264+0000 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:44:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:44:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:44:51.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-30T11:44:53.268 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:44:53.268 INFO:tasks.workunit.client.0.smithi003.stdout:hinfo_key 2024-03-30T11:44:53.268 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:44:53.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:44:53.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:44:53.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:44:53.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:44:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:44:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:44:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:44:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.9 2024-03-30T11:44:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:44:53.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:44:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:44:53.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:44:53.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:44:53.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:44:53.577 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:53.584+0000 7f4ff621e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:53.579 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:53.586+0000 7f4ff621e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:53.580 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:53.587+0000 7f4ff621e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:44:53.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:53.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:54.099 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:54.106+0000 7f4ff621e640 -1 Falling back to public interface 2024-03-30T11:44:54.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:55.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:55.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:55.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:44:55.127 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:55.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:55.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:55.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:56.371 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:44:56.378+0000 7f4ff621e640 -1 osd.9 85 log_to_monitors true 2024-03-30T11:44:56.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:56.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:56.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:44:56.422 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:56.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:56.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:56.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:44:57.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:44:57.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:44:57.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:44:57.788 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:57.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:44:57.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,85) [v2:127.0.0.1:6874/2450215989,v1:127.0.0.1:6875/2450215989] [v2:127.0.0.1:6876/2450215989,v1:127.0.0.1:6877/2450215989] exists,up 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:44:58.326 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:44:58.327 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:44:58.327 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:44:58.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:44:58.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:44:58.327 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:44:58.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:44:58.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:44:58.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:44:58.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:44:58.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:44:58.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:44:58.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:4 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:5 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:6 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:7 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:8 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:9' 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:58.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:44:58.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836564 2024-03-30T11:44:58.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836564 2024-03-30T11:44:58.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564' 2024-03-30T11:44:58.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:58.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:44:59.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-03-30T11:44:59.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-03-30T11:44:59.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038' 2024-03-30T11:44:59.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:44:59.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-03-30T11:44:59.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-03-30T11:44:59.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512' 2024-03-30T11:44:59.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:44:59.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678031 2024-03-30T11:44:59.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678031 2024-03-30T11:44:59.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031' 2024-03-30T11:44:59.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.334 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:44:59.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182461 2024-03-30T11:44:59.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182461 2024-03-30T11:44:59.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031 4-107374182461' 2024-03-30T11:44:59.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:44:59.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285575 2024-03-30T11:44:59.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285575 2024-03-30T11:44:59.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031 4-107374182461 5-356482285575' 2024-03-30T11:44:59.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:44:59.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-03-30T11:44:59.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-03-30T11:44:59.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031 4-107374182461 5-356482285575 6-150323855409' 2024-03-30T11:44:59.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:44:59.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691878 2024-03-30T11:44:59.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691878 2024-03-30T11:44:59.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031 4-107374182461 5-356482285575 6-150323855409 7-171798691878' 2024-03-30T11:44:59.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:44:59.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528353 2024-03-30T11:44:59.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528353 2024-03-30T11:44:59.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031 4-107374182461 5-356482285575 6-150323855409 7-171798691878 8-193273528353' 2024-03-30T11:44:59.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:44:59.860 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:44:59.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-30T11:44:59.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-30T11:44:59.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673038 2-64424509512 3-304942678031 4-107374182461 5-356482285575 6-150323855409 7-171798691878 8-193273528353 9-377957122050' 2024-03-30T11:44:59.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:44:59.966 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836564 2024-03-30T11:44:59.966 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:44:59.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:44:59.967 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836564 2024-03-30T11:44:59.967 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:44:59.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836564 2024-03-30T11:44:59.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-03-30T11:44:59.969 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836564 2024-03-30T11:44:59.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:00.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836564 2024-03-30T11:45:00.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:45:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:45:01.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:01.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836564 2024-03-30T11:45:01.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:01.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-03-30T11:45:01.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:01.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:01.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-03-30T11:45:01.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:01.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-03-30T11:45:01.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-03-30T11:45:01.549 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949673038 2024-03-30T11:45:01.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:01.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673038 2024-03-30T11:45:01.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:01.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-03-30T11:45:01.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:01.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:01.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-03-30T11:45:01.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:01.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-03-30T11:45:01.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-03-30T11:45:01.837 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509512 2024-03-30T11:45:01.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:02.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509513 -lt 64424509512 2024-03-30T11:45:02.146 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:02.146 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678031 2024-03-30T11:45:02.146 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:02.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:02.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678031 2024-03-30T11:45:02.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:02.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678031 2024-03-30T11:45:02.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678031' 2024-03-30T11:45:02.149 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.3 seq 304942678031 2024-03-30T11:45:02.149 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:02.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678032 -lt 304942678031 2024-03-30T11:45:02.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:02.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182461 2024-03-30T11:45:02.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:02.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:02.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182461 2024-03-30T11:45:02.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:02.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182461 2024-03-30T11:45:02.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-03-30T11:45:02.434 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.4 seq 107374182461 2024-03-30T11:45:02.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:02.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182461 -lt 107374182461 2024-03-30T11:45:02.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:02.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285575 2024-03-30T11:45:02.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:02.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:02.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285575 2024-03-30T11:45:02.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:02.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285575 2024-03-30T11:45:02.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285575' 2024-03-30T11:45:02.720 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.5 seq 356482285575 2024-03-30T11:45:02.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:03.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285576 -lt 356482285575 2024-03-30T11:45:03.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:03.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-03-30T11:45:03.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:03.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:03.006 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-03-30T11:45:03.006 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-03-30T11:45:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-03-30T11:45:03.007 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.6 seq 150323855409 2024-03-30T11:45:03.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:03.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855409 -lt 150323855409 2024-03-30T11:45:03.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:03.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691878 2024-03-30T11:45:03.294 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:03.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:03.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691878 2024-03-30T11:45:03.295 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:03.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691878 2024-03-30T11:45:03.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-03-30T11:45:03.296 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.7 seq 171798691878 2024-03-30T11:45:03.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:03.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691878 2024-03-30T11:45:03.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:03.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528353 2024-03-30T11:45:03.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:03.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:03.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528353 2024-03-30T11:45:03.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:03.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528353 2024-03-30T11:45:03.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2024-03-30T11:45:03.582 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.8 seq 193273528353 2024-03-30T11:45:03.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:03.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528354 -lt 193273528353 2024-03-30T11:45:03.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:03.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-03-30T11:45:03.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:03.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:03.878 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-03-30T11:45:03.878 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:03.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-30T11:45:03.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-03-30T11:45:03.880 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.9 seq 377957122050 2024-03-30T11:45:03.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:04.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-03-30T11:45:04.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:04.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:04.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:04.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:04.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:04.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:04.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:04.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:04.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:04.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:04.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:04.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:04.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:04.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:04.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:05.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:05.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:05.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:05.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:45:05.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:45:05.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-30T11:45:05.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:45:05.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 268120"' 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 268120' 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 268122"' 2024-03-30T11:45:05.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 268122' 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 268120 268122' 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 268120 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/268123: /' 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:45:05.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/268125: /' 2024-03-30T11:45:08.552 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.552 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:45:08.553 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:45:08.554 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:45:08.556 INFO:tasks.workunit.client.0.smithi003.stderr:268125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: start osd.9 2024-03-30T11:45:08.557 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:45:08.603 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:45:08.604 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:45:08.605 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:45:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:45:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:45:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:45:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:45:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:45:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:268123: start osd.5 2024-03-30T11:45:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:45:13.322 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:45:13.322 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2024-03-30T11:45:08.641+0000 7f8d5ceae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2024-03-30T11:45:08.643+0000 7f8d5ceae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2024-03-30T11:45:08.644+0000 7f8d5ceae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:45:13.323 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 0 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2024-03-30T11:45:09.164+0000 7f8d5ceae640 -1 Falling back to public interface 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 1 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:45:13.324 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2024-03-30T11:45:12.039+0000 7f8d5ceae640 -1 osd.5 90 log_to_monitors true 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 3 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268123: osd.5 up in weight 1 up_from 96 up_thru 83 down_at 91 last_268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:45:13.325 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2024-03-30T11:45:08.592+0000 7f7a753bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2024-03-30T11:45:08.594+0000 7f7a753bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2024-03-30T11:45:08.595+0000 7f7a753bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 0 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.326 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2024-03-30T11:45:09.114+0000 7f7a753bd640 -1 Falling back to public interface 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 1 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2 2024-03-30T11:45:13.327 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2024-03-30T11:45:11.998+0000 7f7a753bd640 -1 osd.9 90 log_to_monitors true 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 3 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:13.328 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:14.385 INFO:tasks.workunit.client.0.smithi003.stderr:268125: osd.9 up in weight 1 up_from 95 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6842/1679491525,v1:127.0.0.1:6843/1679491525] [v2:127.0.0.1:6844/1679491525,v1:127.0.0.1:6845/1679491525] exists,up 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:45:14.385 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:45:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 1 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 2 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 3 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 4 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 5 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 6 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 7 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 8 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: 9' 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:45:14.387 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673042 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673042 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042' 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qaclean_interval [83,90) [v2:127.0.0.1:6874/2056264837,v1:127.0.0.1:6875/2056264837] [v2:127.0.0.1:6876/2056264837,v1:127.0.0.1:6877/2056264837] exists,up 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:45:14.388 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:45:14.389 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:45:14.389 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:45:14.389 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:45:14.389 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:45:14.390 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 1 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 2 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 3 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 4 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 5 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 6 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 7 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 8 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: 9' 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:14.391 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043' 2024-03-30T11:45:14.392 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:45:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-03-30T11:45:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-03-30T11:45:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517' 2024-03-30T11:45:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.745 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678037 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678037 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037' 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182465 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182465 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037 4-107374182465' 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037 4-107374182465 5-412316860419' 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.746 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855414 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855414 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037 4-107374182465 5-412316860419 6-150323855414' 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037 4-107374182465 5-412316860419 6-150323855414 7-171798691883' 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clon/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516' 2024-03-30T11:45:15.747 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678036 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678036 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036' 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182466 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182466 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036 4-107374182466' 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036 4-107374182466 5-412316860418' 2024-03-30T11:45:15.748 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036 4-107374182466 5-412316860418 6-150323855413' 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036 4-107374182466 5-412316860418 6-150323855413 7-171798691882' 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:45:16.430 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clonee.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-03-30T11:45:16.430 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-03-30T11:45:16.430 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037 4-107374182465 5-412316860419 6-150323855414 7-171798691883 8-193273528358' 2024-03-30T11:45:16.430 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509517 3-304942678037 4-107374182465 5-412316860419 6-150323855414 7-171798691883 8-193273528358 9-408021893122' 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:16.431 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.0 seq 21474836569 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836569 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.1 seq 42949673043 2024-03-30T11:45:16.432 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:16.433 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-03-30T11:45:16.433 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:16.433 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-03-30T11:45:16.433 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:16.433 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:16.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone..client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528357 2024-03-30T11:45:16.513 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528357 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036 4-107374182466 5-412316860418 6-150323855413 7-171798691882 8-193273528357' 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893123 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893123 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673042 2-64424509516 3-304942678036 4-107374182466 5-412316860418 6-150323855413 7-171798691882 8-193273528357 9-408021893123' 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:16.514 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.0 seq 21474836568 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836568 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673042 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673042 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673042 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-03-30T11:45:16.515 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.1 seq 42949673042 2024-03-30T11:45:16.516 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:16.516 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673042 2024-03-30T11:45:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-03-30T11:45:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:16.517 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.cclient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.2 seq 64424509517 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:17.309 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509517 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678037 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678037 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678037 2024-03-30T11:45:17.310 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.3 seq 304942678037 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678037 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182465 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182465 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182465 2024-03-30T11:45:17.311 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.4 seq 107374182465 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182465 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860419 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860419 2024-03-30T11:45:17.312 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.313 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.2 seq 64424509516 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509516 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678036 2024-03-30T11:45:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678036 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678036 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678036' 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.3 seq 304942678036 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678036 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182466 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182466 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182466 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182466' 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.4 seq 107374182466 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182466 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-412316860418 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:17.462 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:17.463 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-412316860418 2024-03-30T11:45:17.463 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:17.463 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-03-30T11:45:18.512 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitiing osd.5 seq 412316860419' 2024-03-30T11:45:18.512 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.5 seq 412316860419 2024-03-30T11:45:18.512 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:18.512 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855414 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855414 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855414 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.6 seq 150323855414 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855414 -lt 150323855414 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:18.513 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.7 seq 171798691883 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691883 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-03-30T11:45:18.514 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-03-30T11:45:18.515 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.8 seq 193273528358 2024-03-30T11:45:18.515 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:18.515 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528358 2024-03-30T11:45:18.662 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_png osd.5 seq 412316860418' 2024-03-30T11:45:18.662 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.5 seq 412316860418 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.6 seq 150323855413 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855414 -lt 150323855413 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:18.663 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.7 seq 171798691882 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691882 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528357 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528357 2024-03-30T11:45:18.664 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:18.665 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528357 2024-03-30T11:45:18.665 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-03-30T11:45:18.665 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.8 seq 193273528357 2024-03-30T11:45:18.665 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:18.665 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528357 2024-03-30T11:45:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgg_stats: for s in $seqs 2024-03-30T11:45:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893122 2024-03-30T11:45:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893122 2024-03-30T11:45:19.961 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893122' 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: waiting osd.9 seq 408021893122 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893123 -lt 408021893122 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:20.499 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ub_stats: for s in $seqs 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-408021893123 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-408021893123 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893123 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 408021893123' 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: waiting osd.9 seq 408021893123 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893123 2024-03-30T11:45:20.500 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:20.501 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:20.502 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:21.549 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:45:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:21.551 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:21.698 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dumpntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:45:21.699 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:21.700 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:22.192 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:22.192 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:22.192 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:268123: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:268123: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 268122 2024-03-30T11:45:22.359 INFO:tasks.workunit.client.0.smithi003.stderr:pgs 2024-03-30T11:45:22.359 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:22.359 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:22.359 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:22.359 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:22.359 INFO:tasks.workunit.client.0.smithi003.stderr:268125: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:268125: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:45:22.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:45:22.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:22.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:22.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:22.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:45:23.196 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:45:23.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:23.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:45:23.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:23.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:23.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:23.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:45.752801+0000 '>' 2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:23.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:45:24.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:45:24.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:24.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:24.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:24.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:24.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:24.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:45.752801+0000 '>' 2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:24.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:45:25.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:45:25.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:25.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:25.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:25.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:25.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:25.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:26.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:44:45.752801+0000 '>' 2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:26.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:45:27.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:45:27.076 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:27.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:27.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:27.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:27.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:27.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:45:23.679863+0000 '>' 2024-03-30T11:44:45.752801+0000 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 271745"' 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 271745' 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:45:27.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 271747"' 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 271747' 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 271745 271747' 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 271745 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/271748: /' 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:45:27.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/271750: /' 2024-03-30T11:45:30.007 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.007 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:45:30.008 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: _ 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: hinfo_key 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: snapset 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:45:30.009 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-30T11:45:30.010 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:45:30.010 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-30T11:45:30.010 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:45:30.010 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:45:30.010 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:30.011 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-30T11:45:30.012 INFO:tasks.workunit.client.0.smithi003.stderr:271750: start osd.9 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:45:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: _ 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: hinfo_key 2024-03-30T11:45:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:271748: snapset 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:45:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:45:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:45:30.143 INFO:tasks.workunit.client.0.smithi003.stderr:271748: start osd.5 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:45:34.546 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2024-03-30T11:45:30.046+0000 7fcfe8136640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2024-03-30T11:45:30.049+0000 7fcfe8136640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2024-03-30T11:45:30.050+0000 7fcfe8136640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 0 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2024-03-30T11:45:30.574+0000 7fcfe8136640 -1 Falling back to public interface 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:34.547 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 1 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2024-03-30T11:45:33.073+0000 7fcfe8136640 -1 osd.9 98 log_to_monitors true 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:34.548 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.549 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:45:34.549 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 3 2024-03-30T11:45:34.549 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.549 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-30T11:45:34.838 INFO:tasks.workunit.client.0.smithi003.stderr:271750: osd.9 up in weight 1 up_from 103 up_thru 0 down_aovery_ops 2024-03-30T11:45:34.838 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:45:34.838 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:45:34.838 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2024-03-30T11:45:30.174+0000 7fb817cce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2024-03-30T11:45:30.176+0000 7fb817cce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2024-03-30T11:45:30.177+0000 7fb817cce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 0 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.839 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2024-03-30T11:45:30.691+0000 7fb817cce640 -1 Falling back to public interface 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 1 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.840 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2024-03-30T11:45:33.423+0000 7fb817cce640 -1 osd.5 98 log_to_monitors true 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 3 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:34.841 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271748: osd.5 up in weight 1 up_from 103 up_thru 103 downt 99 last_clean_interval [95,98) [v2:127.0.0.1:6842/295748195,v1:127.0.0.1:6843/295748195] [v2:127.0.0.1:6844/295748195,v1:127.0.0.1:6845/295748195] exists,up 53b6c8e8-14ff-41f7-b0be-ac0444cedf48 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:45:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:45:36.015 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 1 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 2 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 3 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 4 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 5 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 6 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 7 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 8 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: 9' 2024-03-30T11:45:36.016 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673048 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673048 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048' 2024-03-30T11:45:36.017 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.clie_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6874/797946019,v1:127.0.0.1:6875/797946019] [v2:127.0.0.1:6876/797946019,v1:127.0.0.1:6877/797946019] exists,up 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:45:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:45:36.019 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 1 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 2 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 3 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 4 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 5 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 6 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 7 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 8 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: 9' 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:45:36.020 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049' 2024-03-30T11:45:36.021 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.252 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509523 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509523 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523' 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678043 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678043 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043' 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-03-30T11:45:37.253 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043 4-107374182471' 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043 4-107374182471 5-442381631491' 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043 4-107374182471 5-442381631491 6-150323855420' 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:45:37.254 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691889 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043 4-107374182471 5-442381631491 6-150323855420 7-171798691889' 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtnt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524' 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678042 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678042 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042' 2024-03-30T11:45:37.255 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042 4-107374182472' 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042 4-107374182472 5-442381631490' 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-03-30T11:45:37.256 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042 4-107374182472 5-442381631490 6-150323855419' 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691888 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691888 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042 4-107374182472 5-442381631490 6-150323855419 7-171798691888' 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.257 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:45:37.985 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528364 2024-03-30T11:45:37.985 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528364 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042 4-107374182472 5-442381631490 6-150323855419 7-171798691888 8-193273528364' 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631491 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631491 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673048 2-64424509524 3-304942678042 4-107374182472 5-442381631490 6-150323855419 7-171798691888 8-193273528364 9-442381631491' 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:37.986 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.0 seq 21474836575 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836575 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673048 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673048 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673048 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.1 seq 42949673048 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673049 -lt 42949673048 2024-03-30T11:45:37.987 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/est/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528363 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528363 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043 4-107374182471 5-442381631491 6-150323855420 7-171798691889 8-193273528363' 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-03-30T11:45:37.988 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673049 2-64424509523 3-304942678043 4-107374182471 5-442381631491 6-150323855420 7-171798691889 8-193273528363 9-442381631490' 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.0 seq 21474836576 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:37.989 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.1 seq 42949673049 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673049 -lt 42949673049 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509523 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:37.990 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:38.865 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtesclone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-03-30T11:45:38.865 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.2 seq 64424509524 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509524 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678042 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678042 2024-03-30T11:45:38.866 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678042 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.3 seq 304942678042 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678042 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-03-30T11:45:38.867 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.4 seq 107374182472 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182472 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-442381631490 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-442381631490 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.868 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-03-30T11:45:38.941 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echot/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509523 2024-03-30T11:45:38.941 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.941 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509523 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.2 seq 64424509523 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509523 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678043 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678043 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678043 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.3 seq 304942678043 2024-03-30T11:45:38.942 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678043 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.4 seq 107374182471 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182471 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-442381631491 2024-03-30T11:45:38.943 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-442381631491 2024-03-30T11:45:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-03-30T11:45:40.128 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: ec 'waiting osd.5 seq 442381631490' 2024-03-30T11:45:40.128 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.5 seq 442381631490 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631491 -lt 442381631490 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.6 seq 150323855419 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855420 -lt 150323855419 2024-03-30T11:45:40.129 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691888 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691888 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691888 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.7 seq 171798691888 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691888 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528364 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:40.130 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528364 2024-03-30T11:45:40.131 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:40.131 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528364 2024-03-30T11:45:40.131 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-03-30T11:45:40.131 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.8 seq 193273528364 2024-03-30T11:45:40.131 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:40.131 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528364 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fho 'waiting osd.5 seq 442381631491' 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.5 seq 442381631491 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631491 -lt 442381631491 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:40.210 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.6 seq 150323855420 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855420 -lt 150323855420 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-03-30T11:45:40.211 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.7 seq 171798691889 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691889 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528363 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528363 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528363 2024-03-30T11:45:40.212 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-03-30T11:45:40.213 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.8 seq 193273528363 2024-03-30T11:45:40.213 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:40.213 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528365 -lt 193273528363 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210:lush_pg_stats: for s in $seqs 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631491 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631491 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631491 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631491' 2024-03-30T11:45:41.456 INFO:tasks.workunit.client.0.smithi003.stderr:271750: waiting osd.9 seq 442381631491 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631491 -lt 442381631491 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:41.457 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:271750: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:41.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:45:41.541 INFO:tasks.workunit.client.0.smithi003.stderr: flush_pg_stats: for s in $seqs 2024-03-30T11:45:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631490 2024-03-30T11:45:41.541 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631490 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631490' 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: waiting osd.9 seq 442381631490 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631491 -lt 442381631490 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:41.542 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:271748: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:45:41.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 271747 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:45:41.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:45:41.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 274978"' 2024-03-30T11:45:41.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 274978' 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 274979"' 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 274979' 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 274978 274979' 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 274978 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/274981: /' 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/274983: /' 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:45:44.435 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-30T11:45:44.436 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:45:44.437 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:45:44.438 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:45:44.438 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:45:44.438 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:45:44.438 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:45:44.438 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:45:44.438 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:45:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: start osd.5 2024-03-30T11:45:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:45:44.572 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-30T11:45:44.573 INFO:tasks.workunit.client.0.smithi003.stderr:274981: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:45:44.574 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:44.575 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: start osd.3 2024-03-30T11:45:44.576 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:45:48.931 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2024-03-30T11:45:44.474+0000 7fd00461e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2024-03-30T11:45:44.476+0000 7fd00461e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2024-03-30T11:45:44.477+0000 7fd00461e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:45:48.932 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 0 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2024-03-30T11:45:44.995+0000 7fd00461e640 -1 Falling back to public interface 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 1 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:48.933 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2024-03-30T11:45:47.679+0000 7fd00461e640 -1 osd.5 106 log_to_monitors true 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 3 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:48.934 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:45:49.233 INFO:tasks.workunit.client.0.smithi003.stderr:274983: osd.5 up in weight 1 up_from 110 up_thru 110 down_at 107 l274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2024-03-30T11:45:44.608+0000 7f5327773640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2024-03-30T11:45:44.610+0000 7f5327773640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2024-03-30T11:45:44.611+0000 7f5327773640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:45:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 0 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2024-03-30T11:45:45.123+0000 7f5327773640 -1 Falling back to public interface 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 1 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:49.235 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2024-03-30T11:45:47.853+0000 7f5327773640 -1 osd.3 106 log_to_monitors true 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 3 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:45:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:45:49.881 INFO:tasks.workunit.client.0.smithi003.stderr:274981: osd.3 up in weight 1 up_from 111 up_thru 103 down_at 107 last_clean_interval [103,106) [v2:127.0.0.1:6826/961207551,v1:127.0.0.1:6827/961207551] [v2:127.0.0.1:6828/961207551,v1:127.0.0.1:6829/961207551] exists,up 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:45:49.882 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 1 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 2 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 3 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 4 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 5 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 6 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 7 2024-03-30T11:45:49.883 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 8 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: 9' 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053' 2024-03-30T11:45:49.884 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0ast_clean_interval [71,106) [v2:127.0.0.1:6874/4000280318,v1:127.0.0.1:6875/4000280318] [v2:127.0.0.1:6876/4000280318,v1:127.0.0.1:6877/4000280318] exists,up 31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:45:50.136 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 1 2024-03-30T11:45:50.137 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 2 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 3 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 4 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 5 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 6 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 7 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 8 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: 9' 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.138 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:45:50.139 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-03-30T11:45:50.139 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-03-30T11:45:50.139 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054' 2024-03-30T11:45:50.139 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.clien/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509528 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509528 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528' 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-30T11:45:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858' 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858 4-107374182476' 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858 4-107374182476 5-472446402562' 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:45:50.886 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858 4-107374182476 5-472446402562 6-150323855424' 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691893 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691893 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858 4-107374182476 5-472446402562 6-150323855424 7-171798691893' 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:50.887 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529' 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369859 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369859 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859' 2024-03-30T11:45:51.095 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859 4-107374182477' 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402563 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402563 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859 4-107374182477 5-472446402563' 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-03-30T11:45:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859 4-107374182477 5-472446402563 6-150323855425' 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859 4-107374182477 5-472446402563 6-150323855425 7-171798691894' 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.097 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:45:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtestone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-03-30T11:45:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-03-30T11:45:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858 4-107374182476 5-472446402562 6-150323855424 7-171798691893 8-193273528368' 2024-03-30T11:45:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:45:51.848 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631495 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631495 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673053 2-64424509528 3-476741369858 4-107374182476 5-472446402562 6-150323855424 7-171798691893 8-193273528368 9-442381631495' 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.0 seq 21474836580 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836580 2024-03-30T11:45:51.849 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.1 seq 42949673053 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673053 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509528 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:51.850 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clon/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859 4-107374182477 5-472446402563 6-150323855425 7-171798691894 8-193273528369' 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631496 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631496 2024-03-30T11:45:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673054 2-64424509529 3-476741369859 4-107374182477 5-472446402563 6-150323855425 7-171798691894 8-193273528369 9-442381631496' 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.0 seq 21474836581 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836581 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:51.940 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.1 seq 42949673054 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:51.941 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:45:52.738 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/ce.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509528 2024-03-30T11:45:52.738 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.738 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509528 2024-03-30T11:45:52.738 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.2 seq 64424509528 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509528 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369858 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369858 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369858' 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.3 seq 476741369858 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369858 2024-03-30T11:45:52.739 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.4 seq 107374182476 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182476 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-472446402562 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:52.740 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-472446402562 2024-03-30T11:45:52.741 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.741 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'walone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.2 seq 64424509529 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509529 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-476741369859 2024-03-30T11:45:52.935 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-476741369859 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369859 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 476741369859' 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.3 seq 476741369859 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369859 -lt 476741369859 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-03-30T11:45:52.936 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.4 seq 107374182477 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182477 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-472446402563 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-472446402563 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:52.937 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402563 2024-03-30T11:45:53.914 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo iting osd.5 seq 472446402562' 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.5 seq 472446402562 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402562 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-03-30T11:45:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.6 seq 150323855424 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855424 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691893 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691893 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691893 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691893' 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.7 seq 171798691893 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691893 2024-03-30T11:45:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.8 seq 193273528368 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:53.917 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528368 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush'waiting osd.5 seq 472446402563' 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.5 seq 472446402563 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402563 -lt 472446402563 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:45:54.157 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.6 seq 150323855425 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855425 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.7 seq 171798691894 2024-03-30T11:45:54.158 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.8 seq 193273528369 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:45:54.159 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fl_pg_stats: for s in $seqs 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631495 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631495 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631495 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631495' 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: waiting osd.9 seq 442381631495 2024-03-30T11:45:55.320 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631496 -lt 442381631495 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:55.321 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:55.322 INFO:tasks.workunit.client.0.smithi003.stderr:274983: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:55.322 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:55.322 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:55.322 INFO:tasks.workunit.client.0.smithi003.stderr:274983: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:55.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:ush_pg_stats: for s in $seqs 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631496 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631496 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631496 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631496' 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: waiting osd.9 seq 442381631496 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:45:55.542 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631496 -lt 442381631496 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:45:55.543 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:274981: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:274981: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 274979 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:45:55.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:45:55.545 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-30T11:45:55.545 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:55.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:56.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:45:23.679863+0000 2024-03-30T11:45:56.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T11:45:56.380 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T11:45:23.679863+0000 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:45:23.679863+0000 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:56.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:56.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:45:23.679863+0000 '>' 2024-03-30T11:45:23.679863+0000 2024-03-30T11:45:56.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:45:57.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:45:57.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:57.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:57.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:57.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:57.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:57.672 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:57.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:45:23.679863+0000 '>' 2024-03-30T11:45:23.679863+0000 2024-03-30T11:45:57.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:45:58.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:45:58.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:45:58.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T11:45:58.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T11:45:58.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:45:58.966 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:45:58.966 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T11:45:59.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:45:57.012092+0000 '>' 2024-03-30T11:45:23.679863+0000 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 278334"' 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 278334' 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 278336"' 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 278336' 2024-03-30T11:45:59.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 278334 278336' 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 278334 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/278337: /' 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:45:59.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/278339: /' 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.078 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:46:02.079 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: _ 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: hinfo_key 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: snapset 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-30T11:46:02.080 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:46:02.081 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:46:02.082 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:46:02.083 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:46:02.083 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:46:02.083 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-30T11:46:02.083 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-30T11:46:02.083 INFO:tasks.workunit.client.0.smithi003.stderr:278337: start osd.3 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:46:02.549 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: _ 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: hinfo_key 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: snapset 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:46:02.550 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:46:02.551 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:46:02.552 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:46:02.553 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:46:02.553 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:46:02.553 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-30T11:46:02.553 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-30T11:46:02.553 INFO:tasks.workunit.client.0.smithi003.stderr:278339: start osd.5 2024-03-30T11:46:06.252 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:46:06.252 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-30T11:46:06.252 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-30T11:46:06.252 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:46:06.252 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:46:06.252 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2024-03-30T11:46:02.114+0000 7fec09af9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2024-03-30T11:46:02.117+0000 7fec09af9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2024-03-30T11:46:02.118+0000 7fec09af9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 0 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2024-03-30T11:46:02.641+0000 7fec09af9640 -1 Falling back to public interface 2024-03-30T11:46:06.253 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 1 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:06.254 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2024-03-30T11:46:05.174+0000 7fec09af9640 -1 osd.3 112 log_to_monitors true 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 3 2024-03-30T11:46:06.255 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:07.276 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-30T11:46:07.276 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-30T11:46:07.276 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-30T11:46:07.276 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:46:07.276 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:46:07.276 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2024-03-30T11:46:02.584+0000 7efd72287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2024-03-30T11:46:02.586+0000 7efd72287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2024-03-30T11:46:02.587+0000 7efd72287640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 0 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2024-03-30T11:46:03.109+0000 7efd72287640 -1 Falling back to public interface 2024-03-30T11:46:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 1 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:46:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2024-03-30T11:46:05.926+0000 7efd72287640 -1 osd.5 112 log_to_monitors true 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 3 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:07.279 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278339: osd.5 up in weight 1 up_from 118 up_thru 118 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: osd.3 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [111,112) [v2:127.0.0.1:6826/259380934,v1:127.0.0.1:6827/259380934] [v2:127.0.0.1:6828/259380934,v1:127.0.0.1:6829/259380934] exists,up 31eb0563-7c9d-4921-a411-324fd3c7a7a0 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:46:07.402 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:46:07.403 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 1 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 2 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 3 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 4 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 5 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 6 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 7 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 8 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: 9' 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-03-30T11:46:07.404 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:07.405 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:46:07.405 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-03-30T11:46:07.405 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-03-30T11:46:07.405 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059' 2024-03-30T11:46:08.372 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/n_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6874/2991450640,v1:127.0.0.1:6875/2991450640] [v2:127.0.0.1:6876/2991450640,v1:127.0.0.1:6877/2991450640] exists,up 9b31de6a-d097-42ce-ade3-a33a9bdc1575 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:46:08.373 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 1 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 2 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 3 2024-03-30T11:46:08.374 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 4 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 5 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 6 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 7 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 8 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: 9' 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-03-30T11:46:08.375 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-03-30T11:46:08.376 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.376 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:46:08.376 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-03-30T11:46:08.376 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-03-30T11:46:08.376 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060' 2024-03-30T11:46:08.376 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.856 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.856 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:46:08.856 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-03-30T11:46:08.856 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533' 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338' 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182482 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182482 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338 4-107374182482' 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.857 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338 4-107374182482 5-506806140930' 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855429 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855429 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338 4-107374182482 5-506806140930 6-150323855429' 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-03-30T11:46:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-03-30T11:46:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338 4-107374182482 5-506806140930 6-150323855429 7-171798691899' 2024-03-30T11:46:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/stclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534' 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206339 2024-03-30T11:46:09.395 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206339 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339' 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182483 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182483 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339 4-107374182483' 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339 4-107374182483 5-506806140931' 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-30T11:46:09.396 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855431 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855431 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339 4-107374182483 5-506806140931 6-150323855431' 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339 4-107374182483 5-506806140931 6-150323855431 7-171798691900' 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.397 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:46:09.662 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubunandalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-30T11:46:09.662 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-03-30T11:46:09.662 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-03-30T11:46:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338 4-107374182482 5-506806140930 6-150323855429 7-171798691899 8-193273528374' 2024-03-30T11:46:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:46:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631501 2024-03-30T11:46:09.663 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631501 2024-03-30T11:46:09.664 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673059 2-64424509533 3-498216206338 4-107374182482 5-506806140930 6-150323855429 7-171798691899 8-193273528374 9-442381631501' 2024-03-30T11:46:09.664 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:09.664 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-03-30T11:46:09.664 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:09.664 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:46:09.664 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-03-30T11:46:09.665 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:09.665 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-03-30T11:46:09.665 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-03-30T11:46:09.665 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.0 seq 21474836585 2024-03-30T11:46:09.665 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:46:09.665 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836585 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.1 seq 42949673059 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673059 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:09.666 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-03-30T11:46:09.667 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.160 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtestu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-03-30T11:46:10.160 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-03-30T11:46:10.160 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339 4-107374182483 5-506806140931 6-150323855431 7-171798691900 8-193273528375' 2024-03-30T11:46:10.160 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631502 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631502 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673060 2-64424509534 3-498216206339 4-107374182483 5-506806140931 6-150323855431 7-171798691900 8-193273528375 9-442381631502' 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.0 seq 21474836586 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:46:10.161 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836586 -lt 21474836586 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.1 seq 42949673060 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673060 -lt 42949673060 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.162 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntut/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.2 seq 64424509533 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509533 2024-03-30T11:46:10.588 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206338 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206338 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206338' 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.3 seq 498216206338 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206338 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182482 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:46:10.589 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182482 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182482 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.4 seq 107374182482 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182482 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140930 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140930 2024-03-30T11:46:10.590 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.116 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=50680614/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-03-30T11:46:11.116 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.116 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-03-30T11:46:11.116 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.2 seq 64424509534 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509534 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206339 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206339 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206339 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206339' 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.3 seq 498216206339 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-30T11:46:11.117 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206339 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182483 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182483 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182483 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.4 seq 107374182483 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182483 -lt 107374182483 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-506806140931 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:11.118 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-30T11:46:11.119 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-506806140931 2024-03-30T11:46:11.119 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.119 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-03-30T11:46:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_st0930 2024-03-30T11:46:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 506806140930' 2024-03-30T11:46:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.5 seq 506806140930 2024-03-30T11:46:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:46:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140930 -lt 506806140930 2024-03-30T11:46:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855429 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855429 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855429 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.6 seq 150323855429 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855431 -lt 150323855429 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:46:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.7 seq 171798691899 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691899 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-03-30T11:46:11.812 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-03-30T11:46:11.813 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.8 seq 193273528374 2024-03-30T11:46:11.813 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:46:12.364 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 19327ats: echo 'waiting osd.5 seq 506806140931' 2024-03-30T11:46:12.364 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.5 seq 506806140931 2024-03-30T11:46:12.364 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140931 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855431 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855431 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855431 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.6 seq 150323855431 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855431 -lt 150323855431 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:12.365 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.7 seq 171798691900 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691900 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-03-30T11:46:12.366 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:12.367 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-03-30T11:46:12.367 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-03-30T11:46:12.367 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.8 seq 193273528375 2024-03-30T11:46:12.367 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-30T11:46:12.367 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528375 -lt 193273528375 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.3528375 -lt 193273528374 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631501 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631501 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631501 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631501' 2024-03-30T11:46:13.172 INFO:tasks.workunit.client.0.smithi003.stderr:278337: waiting osd.9 seq 442381631501 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631501 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:46:13.173 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:278337: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:278337: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-30T11:46:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 278336 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-442381631502 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-442381631502 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631502 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 442381631502' 2024-03-30T11:46:13.694 INFO:tasks.workunit.client.0.smithi003.stderr:278339: waiting osd.9 seq 442381631502 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631502 -lt 442381631502 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T11:46:13.695 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:278339: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:278339: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-30T11:46:13.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:46:13.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:46:13.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:46:13.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:46:13.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:46:13.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:46:13.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:46:13.862 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:46:13.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:46:13.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:46:13.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:46:13.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:46:13.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:46:13.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:46:13.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:46:13.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:46:13.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:46:13.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:46:13.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:46:13.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:13.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:13.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:46:13.928 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:46:13.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:46:13.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:46:13.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:46:13.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:46:13.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:46:13.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:46:13.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:46:13.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:46:13.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:46:13.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:46:13.932 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:46:13.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:46:13.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:46:13.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:46:13.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:46:13.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:46:13.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:46:13.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:46:13.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:46:13.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:13.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:13.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:46:13.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:46:13.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:46:13.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:46:13.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:46:13.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:13.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:13.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:46:13.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:46:13.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:46:13.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:46:13.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:46:13.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-30T11:46:14.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:46:14.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:46:14.329 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:46:14.330 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:46:14.330 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:14.330 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:14.330 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:14.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-30T11:46:14.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:46:14.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:46:14.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:46:14.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:46:14.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:46:14.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:46:14.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:46:14.608 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:46:14.609 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:14.609 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:14.609 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:46:14.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:46:14.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:46:14.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-03-30T11:46:14.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:46:14.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:46:14.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:46:14.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:46:14.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:46:14.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:46:15.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:15.508 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:46:15.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:46:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-03-30T11:46:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:46:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:46:15.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:46:15.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:46:15.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:46:15.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:46:15.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:46:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a39368ac-0e69-4b0c-b120-d4e961665a18 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a39368ac-0e69-4b0c-b120-d4e961665a18' 2024-03-30T11:46:15.545 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 a39368ac-0e69-4b0c-b120-d4e961665a18 2024-03-30T11:46:15.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:46:15.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCH+wdmgGS0IRAAQHstNgVyzNWMHZX4N6IRIg== 2024-03-30T11:46:15.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCH+wdmgGS0IRAAQHstNgVyzNWMHZX4N6IRIg=="}' 2024-03-30T11:46:15.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a39368ac-0e69-4b0c-b120-d4e961665a18 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:46:15.762 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:46:15.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:46:15.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCH+wdmgGS0IRAAQHstNgVyzNWMHZX4N6IRIg== --osd-uuid a39368ac-0e69-4b0c-b120-d4e961665a18 2024-03-30T11:46:15.801 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.808+0000 7f94b181c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:15.813 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.820+0000 7f94b181c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:15.814 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.821+0000 7f94b181c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:15.815 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.822+0000 7f94b181c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:46:15.815 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.822+0000 7f94b181c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:46:15.815 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.822+0000 7f94b181c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:46:15.897 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:15.904+0000 7f94b181c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:46:21.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:46:21.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:46:21.493 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:46:21.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:46:21.494 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:46:21.897 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:46:21.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:46:21.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:46:21.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:46:21.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:46:21.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:46:21.923 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:21.930+0000 7f847121e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:21.925 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:21.932+0000 7f847121e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:21.926 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:21.933+0000 7f847121e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:22.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:46:22.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:46:22.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:46:22.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:46:22.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:46:22.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:22.179 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:46:22.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:46:22.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:22.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:22.441 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:22.448+0000 7f847121e640 -1 Falling back to public interface 2024-03-30T11:46:22.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:23.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:23.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:23.462 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:46:23.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:46:23.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:23.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:24.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:24.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:24.753 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:46:24.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:46:24.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:24.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:24.777 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:24.784+0000 7f847121e640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:46:25.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:26.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:26.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:26.204 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:46:26.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:46:26.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:26.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:26.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:27.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:27.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:27.498 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:46:27.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:46:27.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:27.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:27.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:28.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:28.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:28.792 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:46:28.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:46:28.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:28.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:29.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:30.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:30.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:30.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:46:30.085 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:46:30.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:30.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:30.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:31.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:31.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:31.378 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:46:31.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:46:31.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:31.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:31.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:32.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:32.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:32.723 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:46:32.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:46:32.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:32.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:33.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:34.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:34.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:34.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:46:34.018 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:46:34.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:34.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:34.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:35.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:35.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:35.323 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:46:35.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:46:35.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:35.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:35.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:36.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:36.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:36.671 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:46:36.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:46:36.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:36.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:37.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:38.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:38.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:38.158 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:46:38.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:46:38.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:38.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:38.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:39.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:39.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:39.900 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:46:39.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:46:39.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:39.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:40.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:41.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:46:41.795 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:46:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:41.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:42.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:43.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:43.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:43.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:46:43.711 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:46:43.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:43.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:44.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:45.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:45.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:45.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:46:45.124 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:46:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:45.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:45.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:46.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:46.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:46.596 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:46:46.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:46:46.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:46.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:46:46.919 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1183304763,v1:127.0.0.1:6803/1183304763] [v2:127.0.0.1:6804/1183304763,v1:127.0.0.1:6805/1183304763] exists,up a39368ac-0e69-4b0c-b120-d4e961665a18 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:46:46.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:46:46.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:46:46.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:46:46.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:46:46.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9' 2024-03-30T11:46:46.926 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:46:46.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:46:46.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCm+wdmBTFyOBAAhEw/2LPsp5bNyjimreUv+g== 2024-03-30T11:46:46.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCm+wdmBTFyOBAAhEw/2LPsp5bNyjimreUv+g=="}' 2024-03-30T11:46:46.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:46:47.350 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:46:47.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:46:47.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCm+wdmBTFyOBAAhEw/2LPsp5bNyjimreUv+g== --osd-uuid 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:46:47.389 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.396+0000 7fb354181640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:47.434 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.441+0000 7fb354181640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:47.435 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.442+0000 7fb354181640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:47.436 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.443+0000 7fb354181640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:46:47.436 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.443+0000 7fb354181640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:46:47.436 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.443+0000 7fb354181640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:46:47.501 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:47.508+0000 7fb354181640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:46:53.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:46:53.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:46:53.547 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:46:53.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:46:53.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:46:53.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:46:53.915 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:46:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:46:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:46:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:46:53.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:46:53.942 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:53.949+0000 7f31f9b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:53.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:53.951+0000 7f31f9b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:53.945 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:53.952+0000 7f31f9b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:46:54.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:46:54.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:46:54.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:46:54.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:46:54.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:46:54.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:54.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:46:54.201 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:46:54.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:54.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:46:54.461 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:54.468+0000 7f31f9b06640 -1 Falling back to public interface 2024-03-30T11:46:54.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:46:55.485 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:46:55.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:55.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:46:55.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:56.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:56.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:56.778 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:46:56.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:46:56.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:56.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:46:56.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:46:56.890+0000 7f31f9b06640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:46:57.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:58.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:58.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:58.091 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:46:58.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:46:58.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:58.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:46:58.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:46:59.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:46:59.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:46:59.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:46:59.385 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:46:59.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:46:59.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:46:59.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:00.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:00.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:00.695 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:47:00.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:47:00.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:00.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:00.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:01.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:01.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:01.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:47:02.000 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:47:02.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:02.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:02.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:03.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:03.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:03.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:47:03.295 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:47:03.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:03.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:04.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:04.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:04.712 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:47:04.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:47:04.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:04.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:05.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:06.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:06.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:06.010 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:47:06.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:47:06.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:06.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:06.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:07.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:07.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:07.303 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:47:07.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:47:07.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:07.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:07.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:08.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:08.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:08.659 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:47:08.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:47:08.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:08.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:09.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:10.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:10.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:10.229 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:47:10.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:47:10.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:10.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:10.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:11.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:11.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:11.907 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:47:11.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:47:11.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:11.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:12.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:13.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:13.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:13.711 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:47:13.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:47:13.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:13.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:15.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:15.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:15.531 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:47:15.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:47:15.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:15.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:16.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:17.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:17.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:17.133 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:47:17.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:47:17.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:17.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:17.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:18.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:18.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:18.642 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:47:18.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:47:18.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:18.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:19.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:20.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:20.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:20.559 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:47:20.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:47:20.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:20.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:20.894 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/321889665,v1:127.0.0.1:6811/321889665] [v2:127.0.0.1:6812/321889665,v1:127.0.0.1:6813/321889665] exists,up 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:47:20.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:47:20.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:47:20.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:47:20.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-03-30T11:47:20.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:47:21.166 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:47:21.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:47:21.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:47:21.579 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:47:21.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:47:22.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:47:24.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-03-30T11:47:24.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:47:24.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:47:24.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:47:24.615 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:47:24.615 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:47:24.615 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:47:24.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:47:24.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:47:24.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:47:24.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:47:24.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:47:24.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:47:24.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:47:24.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:47:24.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:47:25.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:47:25.000 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:47:25.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:47:25.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:47:25.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:47:25.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-03-30T11:47:25.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-03-30T11:47:25.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-03-30T11:47:25.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:47:25.104 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:47:25.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-30T11:47:25.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-30T11:47:25.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672967' 2024-03-30T11:47:25.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:47:25.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-03-30T11:47:25.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:47:25.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:47:25.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-03-30T11:47:25.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:47:25.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-03-30T11:47:25.213 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836492 2024-03-30T11:47:25.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-03-30T11:47:25.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:25.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-30T11:47:25.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:47:26.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:47:26.496 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:26.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-03-30T11:47:26.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:47:26.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-30T11:47:26.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:47:26.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:47:26.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-30T11:47:26.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:47:26.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-30T11:47:26.794 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672967 2024-03-30T11:47:26.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-30T11:47:26.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:47:27.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-03-30T11:47:27.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:47:27.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:47:27.075 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:47:27.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:47:27.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:47:27.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:47:27.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:47:27.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:47:27.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:47:28.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:47:28.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:47:28.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:47:28.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-03-30T11:47:28.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:47:28.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-30T11:47:28.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-30T11:47:28.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:47:28.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:47:28.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:47:28.433 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:47:28.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:47:28.935 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:47:28.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:47:28.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:47:28.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-30T11:47:29.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-30T11:47:29.232 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:47:29.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T11:47:29.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-30T11:47:29.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:47:29.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:47:29.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:47:30.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:47:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-30T11:47:31.639 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:47:32.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:47:32.157 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:47:32.158 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:47:32.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:47:32.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:47:32.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:47:32.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:47:32.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:47:32.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:47:32.185 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:32.193+0000 7f084c621640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:32.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:32.195+0000 7f084c621640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:32.189 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:32.196+0000 7f084c621640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:47:32.451 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:47:32.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:32.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:47:32.715 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:32.722+0000 7f084c621640 -1 Falling back to public interface 2024-03-30T11:47:32.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:33.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:33.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:33.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:47:33.739 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:47:33.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:33.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:47:34.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:47:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:47:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:47:35.146 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:35.153+0000 7f084c621640 -1 osd.0 21 log_to_monitors true 2024-03-30T11:47:35.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:36.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:36.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:36.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:47:36.377 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:47:36.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:36.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2842346808,v1:127.0.0.1:6803/2842346808] [v2:127.0.0.1:6804/2842346808,v1:127.0.0.1:6805/2842346808] exists,up a39368ac-0e69-4b0c-b120-d4e961665a18 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:47:36.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:47:36.803 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:47:36.803 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:47:36.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:47:36.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:47:36.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:47:36.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:47:36.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:47:36.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:47:36.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:47:36.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:47:36.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:47:37.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:47:37.170 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:47:37.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:47:37.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:47:37.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:47:37.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-30T11:47:37.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-30T11:47:37.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-30T11:47:37.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:47:37.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:47:37.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-30T11:47:37.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-30T11:47:37.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672971' 2024-03-30T11:47:37.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:47:37.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-30T11:47:37.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:47:37.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:47:37.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-30T11:47:37.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:47:37.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-30T11:47:37.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-30T11:47:37.384 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 111669149698 2024-03-30T11:47:37.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:37.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-30T11:47:37.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:47:38.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:47:38.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:38.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-30T11:47:38.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:47:38.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-30T11:47:38.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:47:38.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:47:38.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-30T11:47:38.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:47:38.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-30T11:47:38.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-30T11:47:38.965 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672971 2024-03-30T11:47:38.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:47:39.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-30T11:47:39.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:47:39.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:47:39.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:47:39.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:47:39.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:47:39.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:47:39.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:47:39.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:47:39.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:47:39.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:47:39.604 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:47:39.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:47:39.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:47:39.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:47:39.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:47:40.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:47:40.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:47:40.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:47:40.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-30T11:47:40.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-30T11:47:40.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:47:40.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-30T11:47:40.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:47:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-30T11:47:40.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-30T11:47:40.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-30T11:47:40.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-30T11:47:40.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:47:40.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:47:40.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:47:40.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:47:40.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:40.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-30T11:47:41.063 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-30T11:47:41.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:41.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-30T11:47:41.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:41.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:47:41.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:47:41.075 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:47:41.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:47:41.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:47:41.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:47:41.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:47:41.076 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:47:41.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:21.526260+0000 '>' 2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:41.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:47:42.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:47:42.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:47:42.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:47:42.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:47:42.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:47:42.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:47:42.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:47:42.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:21.526260+0000 '>' 2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:42.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:47:43.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:47:43.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:47:43.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:47:43.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:47:43.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:47:43.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:47:43.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:47:43.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:21.526260+0000 '>' 2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:43.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:47:44.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:47:44.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:47:44.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:47:44.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:47:44.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:47:44.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:47:44.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:47:45.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:21.526260+0000 '>' 2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:45.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:47:46.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:47:46.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:47:46.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:47:46.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:47:46.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:47:46.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:47:46.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:47:46.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:41.974291+0000 '>' 2024-03-30T11:47:21.526260+0000 2024-03-30T11:47:46.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:47:46.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:47:46.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:47:46.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:47:46.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:47:46.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:47:46.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:47:46.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:47:46.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:47:46.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:47:46.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:47:46.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:47:46.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:47:46.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:47:47.970 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:47:47.970 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:47:47.970 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:47:48.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:47:48.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:47:48.240 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:47:48.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:47:48.241 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:47:48.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:47:48.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:47:48.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:47:48.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:47:48.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:48.275+0000 7f8f0f662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:48.270 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:48.277+0000 7f8f0f662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:48.271 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:48.278+0000 7f8f0f662640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:48.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:47:48.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:47:48.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:48.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:48.791 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:48.798+0000 7f8f0f662640 -1 Falling back to public interface 2024-03-30T11:47:48.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:49.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:49.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:49.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:47:49.816 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:47:49.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:49.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:50.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:47:51.008 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:51.014+0000 7f8f0f662640 -1 osd.1 28 log_to_monitors true 2024-03-30T11:47:51.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:47:51.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:51.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:47:51.108 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:47:51.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:51.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 32 up_thru 27 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2204870302,v1:127.0.0.1:6811/2204870302] [v2:127.0.0.1:6812/2204870302,v1:127.0.0.1:6813/2204870302] exists,up 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:47:51.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:47:51.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:47:51.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:47:51.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:47:51.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:47:51.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:47:51.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:47:51.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:47:51.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:47:51.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:47:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:47:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:47:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:47:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:47:51.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:47:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-30T11:47:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-30T11:47:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-03-30T11:47:52.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:47:52.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:47:52.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-30T11:47:52.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-30T11:47:52.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-03-30T11:47:52.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:47:52.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-03-30T11:47:52.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:47:52.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:47:52.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-03-30T11:47:52.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:47:52.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-30T11:47:52.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-03-30T11:47:52.216 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 111669149702 2024-03-30T11:47:52.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:52.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-03-30T11:47:52.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:47:53.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:47:53.501 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:53.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-03-30T11:47:53.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:47:54.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:47:54.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:47:55.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-30T11:47:55.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:47:55.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-30T11:47:55.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:47:55.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:47:55.091 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-30T11:47:55.091 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:47:55.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-30T11:47:55.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-30T11:47:55.092 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 137438953474 2024-03-30T11:47:55.093 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:47:55.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-30T11:47:55.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:47:55.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:47:55.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:47:55.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:47:55.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:47:56.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:47:56.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:47:56.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:47:56.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:47:56.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:47:56.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:47:56.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:47:56.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:47:56.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:47:56.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-03-30T11:47:56.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-30T11:47:56.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-30T11:47:56.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-30T11:47:56.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:47:56.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:47:56.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:47:57.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-30T11:47:58.273 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:47:58.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:47:58.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:47:58.793 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:47:58.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:47:58.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:47:58.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:47:58.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:47:58.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:47:58.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:47:58.821 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:58.828+0000 7f6661a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:58.823 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:58.830+0000 7f6661a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:58.824 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:58.831+0000 7f6661a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:47:59.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:47:59.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:47:59.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:47:59.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:47:59.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:47:59.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:47:59.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:47:59.087 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:47:59.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:47:59.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:47:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:47:59.346+0000 7f6661a1b640 -1 Falling back to public interface 2024-03-30T11:47:59.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:00.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:00.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:00.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:48:00.366 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:48:00.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:00.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:48:00.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:01.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:01.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:01.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:48:01.661 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:48:01.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:01.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:48:01.706 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:01.713+0000 7f6661a1b640 -1 osd.1 34 log_to_monitors true 2024-03-30T11:48:02.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:03.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:03.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:03.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:48:03.014 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:48:03.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:03.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/2355759619,v1:127.0.0.1:6811/2355759619] [v2:127.0.0.1:6812/2355759619,v1:127.0.0.1:6813/2355759619] exists,up 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:48:03.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:48:03.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:48:03.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:48:03.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:48:03.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:48:03.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:48:03.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:48:03.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:48:03.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:48:03.673 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:48:03.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:48:03.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:48:03.673 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:48:03.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149705 2024-03-30T11:48:03.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149705 2024-03-30T11:48:03.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149705' 2024-03-30T11:48:03.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:48:03.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:48:03.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-03-30T11:48:03.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-03-30T11:48:03.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149705 1-163208757250' 2024-03-30T11:48:03.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:48:03.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149705 2024-03-30T11:48:03.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:48:03.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:48:03.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149705 2024-03-30T11:48:03.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:48:03.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149705 2024-03-30T11:48:03.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2024-03-30T11:48:03.887 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 111669149705 2024-03-30T11:48:03.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:48:04.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149705 2024-03-30T11:48:04.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:48:05.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:48:05.170 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:48:05.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149705 2024-03-30T11:48:05.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:48:06.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:48:06.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:48:06.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149706 -lt 111669149705 2024-03-30T11:48:06.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:48:06.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-163208757250 2024-03-30T11:48:06.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:48:06.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:48:06.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-163208757250 2024-03-30T11:48:06.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:48:06.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-03-30T11:48:06.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-03-30T11:48:06.766 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 163208757250 2024-03-30T11:48:06.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:48:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-03-30T11:48:07.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:48:07.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:07.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:07.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:48:07.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:48:07.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:48:07.690 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:48:07.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:07.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:08.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:48:08.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:48:08.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:48:08.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-30T11:48:08.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-30T11:48:08.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-30T11:48:08.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-30T11:48:08.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:48:08.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:48:08.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:48:08.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:08.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-30T11:48:08.888 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:48:08.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:48:09.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:41.974291+0000 '>' 2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:09.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:48:10.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:48:10.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:48:10.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:48:10.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:48:10.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:48:10.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:48:10.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:48:10.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:41.974291+0000 '>' 2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:10.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:48:11.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:48:11.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:48:11.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:48:11.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:48:11.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:48:11.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:48:11.472 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:48:11.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:41.974291+0000 '>' 2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:11.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:48:12.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:48:12.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:48:12.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:48:12.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:48:12.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:48:12.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:48:12.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:48:13.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:47:41.974291+0000 '>' 2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:13.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:48:14.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:48:14.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:48:14.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-30T11:48:14.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-30T11:48:14.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:48:14.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:48:14.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:48:09.674253+0000 '>' 2024-03-30T11:47:41.974291+0000 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:48:14.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:48:14.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:48:14.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:48:14.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-30T11:48:14.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:48:14.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:48:14.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:48:14.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:48:14.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:48:14.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-30T11:48:16.040 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-30T11:48:16.041 INFO:tasks.workunit.client.0.smithi003.stdout:_ 2024-03-30T11:48:16.041 INFO:tasks.workunit.client.0.smithi003.stdout:snapset 2024-03-30T11:48:16.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:48:16.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:48:16.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:48:16.312 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:48:16.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:48:16.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:48:16.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:48:16.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:48:16.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:48:16.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:48:16.340 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:16.346+0000 7f680b793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:16.342 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:16.349+0000 7f680b793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:16.344 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:16.351+0000 7f680b793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:48:16.858 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:16.865+0000 7f680b793640 -1 Falling back to public interface 2024-03-30T11:48:16.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:17.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:17.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:17.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:48:17.908 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:48:17.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:17.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:48:18.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:19.141 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:19.148+0000 7f680b793640 -1 osd.1 39 log_to_monitors true 2024-03-30T11:48:19.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:19.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:19.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:48:19.200 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:48:19.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:19.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:48:19.532 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 43 up_thru 38 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/2435606546,v1:127.0.0.1:6811/2435606546] [v2:127.0.0.1:6812/2435606546,v1:127.0.0.1:6813/2435606546] exists,up 5602dbd6-d9d9-4c78-ad8d-4ae001ddb9c9 2024-03-30T11:48:19.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:48:19.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:48:19.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:48:19.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:48:19.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:48:19.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:48:19.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:48:19.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:48:19.533 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:48:19.534 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:48:19.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:48:19.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:48:19.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:48:19.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:48:19.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:48:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:48:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:48:19.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:48:19.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:48:20.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:48:20.005 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:48:20.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:48:20.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:48:20.006 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:48:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149709 2024-03-30T11:48:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149709 2024-03-30T11:48:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709' 2024-03-30T11:48:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:48:20.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:48:20.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-30T11:48:20.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-30T11:48:20.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709 1-184683593730' 2024-03-30T11:48:20.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:48:20.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149709 2024-03-30T11:48:20.338 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:48:20.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:48:20.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149709 2024-03-30T11:48:20.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:48:20.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149709 2024-03-30T11:48:20.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149709' 2024-03-30T11:48:20.340 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 111669149709 2024-03-30T11:48:20.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:48:20.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149708 -lt 111669149709 2024-03-30T11:48:20.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:48:21.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:48:21.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:48:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149710 -lt 111669149709 2024-03-30T11:48:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:48:21.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593730 2024-03-30T11:48:21.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:48:21.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:48:21.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593730 2024-03-30T11:48:21.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:48:21.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-30T11:48:21.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-03-30T11:48:21.944 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 184683593730 2024-03-30T11:48:21.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:48:22.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-30T11:48:22.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:48:22.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:22.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:22.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:48:22.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:48:22.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:48:22.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:48:22.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:22.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:23.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-03-30T11:48:23.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-30T11:48:23.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:48:23.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-30T11:48:23.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:48:23.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:48:23.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:48:23.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:48:23.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:48:23.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:48:23.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:48:23.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:48:23.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:48:23.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:48:23.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:48:23.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:48:23.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:23.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:24.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:48:24.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:48:24.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:48:24.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:48:24.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:48:24.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:48:24.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:48:24.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:48:24.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:48:24.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:48:24.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:24.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:25.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-03-30T11:48:25.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:48:25.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:48:25.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:48:25.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:48:25.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:48:25.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:48:25.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:48:25.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:48:25.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:48:25.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:48:25.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T11:48:25.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:48:25.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T11:48:25.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:48:25.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:48:26.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:48:26.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:48:26.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:48:26.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:48:26.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:48:26.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:48:26.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:48:26.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-30T11:48:26.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-30T11:48:26.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:48:26.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:48:26.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:48:26.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:48:26.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:48:26.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:48:26.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:48:26.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:48:26.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:48:26.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:48:26.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:48:26.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:48:26.733 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:48:26.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:48:26.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:48:26.734 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:48:26.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:48:26.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:48:26.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:48:26.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:48:26.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:48:26.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:48:26.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:48:26.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:48:26.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:26.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:26.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:48:26.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:48:26.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:48:26.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:48:26.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:48:26.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:48:26.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:48:26.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:48:26.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:48:26.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:48:26.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:48:26.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:48:26.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:48:26.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:48:26.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:48:26.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:48:26.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:48:26.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:48:26.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:48:26.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:48:26.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:48:26.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:48:26.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:26.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:48:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:48:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:48:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:48:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:48:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:48:26.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:48:26.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:48:26.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:48:26.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:48:27.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:48:27.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:48:27.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:48:27.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:48:27.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:48:27.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:48:27.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:48:27.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:48:27.315 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:48:27.315 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:48:27.315 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:48:27.315 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:48:27.315 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:27.316 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:27.316 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:48:27.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:48:27.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:48:27.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:48:27.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:48:27.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:48:27.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:48:27.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:48:27.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:48:27.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:48:28.306 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:48:28.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:48:28.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:48:28.307 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:48:28.307 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:28.307 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:28.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:48:28.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:48:28.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:48:28.335 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:48:28.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:48:28.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:48:28.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:48:28.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:48:28.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9' 2024-03-30T11:48:28.341 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:48:28.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:48:28.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAM/AdmQrqyFRAAkse6MoTx2JJbI5gzemCAig== 2024-03-30T11:48:28.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAM/AdmQrqyFRAAkse6MoTx2JJbI5gzemCAig=="}' 2024-03-30T11:48:28.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:48:28.570 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:48:28.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:48:28.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAM/AdmQrqyFRAAkse6MoTx2JJbI5gzemCAig== --osd-uuid 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:48:28.609 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.616+0000 7fc9d280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:28.621 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.628+0000 7fc9d280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:28.622 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.629+0000 7fc9d280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:28.623 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.630+0000 7fc9d280f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:48:28.623 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.630+0000 7fc9d280f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:48:28.623 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.630+0000 7fc9d280f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:48:28.713 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:28.720+0000 7fc9d280f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:48:34.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:48:34.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:48:34.377 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:48:34.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:48:34.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:48:34.751 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:48:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:48:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:48:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:48:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:48:34.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:48:34.778 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:34.785+0000 7f50b0c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:34.780 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:34.787+0000 7f50b0c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:34.781 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:34.788+0000 7f50b0c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:48:35.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:48:35.033 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:48:35.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:35.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:35.304 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:35.311+0000 7f50b0c22640 -1 Falling back to public interface 2024-03-30T11:48:35.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:36.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:36.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:36.313 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:48:36.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:48:36.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:36.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:36.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:37.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:37.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:37.606 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:48:37.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:48:37.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:37.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:37.640 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:48:37.647+0000 7f50b0c22640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:48:37.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:38.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:48:38.944 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:48:38.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:38.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:39.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:40.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:40.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:40.266 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:48:40.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:48:40.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:40.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:40.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:41.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:41.665 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:48:41.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:48:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:41.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:41.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:42.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:42.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:42.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:48:42.965 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:48:42.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:42.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:43.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:44.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:44.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:44.262 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:48:44.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:48:44.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:44.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:44.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:45.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:45.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:45.658 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:48:45.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:48:45.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:45.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:45.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:46.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:46.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:46.952 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:48:46.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:48:46.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:46.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:47.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:48.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:48:48.275 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:48:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:48.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:49.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:49.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:49.617 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:48:49.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:48:49.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:49.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:50.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:51.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:51.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:51.064 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:48:51.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:48:51.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:51.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:51.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:52.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:52.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:52.709 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:48:52.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:48:52.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:52.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:53.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:54.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:54.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:54.486 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:48:54.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:48:54.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:54.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:55.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:56.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:56.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:56.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:48:56.582 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:48:56.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:56.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:57.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:48:58.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:48:58.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:48:58.334 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:48:58.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:48:58.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:48:58.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:48:59.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:00.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:00.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:00.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:49:00.079 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:49:00.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:00.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:49:00.396 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3527354642,v1:127.0.0.1:6803/3527354642] [v2:127.0.0.1:6804/3527354642,v1:127.0.0.1:6805/3527354642] exists,up 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:49:00.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:49:00.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:49:00.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:49:00.403 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:49:00.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e9aebfde-ac33-4166-baad-e5936e49e188' 2024-03-30T11:49:00.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:49:00.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAs/AdmSoJXGRAA4Pi4pDaPzEBz19jUkXCBHQ== 2024-03-30T11:49:00.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAs/AdmSoJXGRAA4Pi4pDaPzEBz19jUkXCBHQ=="}' 2024-03-30T11:49:00.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9aebfde-ac33-4166-baad-e5936e49e188 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:49:00.812 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:49:00.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:49:00.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAs/AdmSoJXGRAA4Pi4pDaPzEBz19jUkXCBHQ== --osd-uuid e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:49:00.851 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.858+0000 7fc915266640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:00.888 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.895+0000 7fc915266640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:00.889 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.896+0000 7fc915266640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:00.889 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.896+0000 7fc915266640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:49:00.889 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.896+0000 7fc915266640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:49:00.889 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.896+0000 7fc915266640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:49:00.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:00.979+0000 7fc915266640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:49:06.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:49:06.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:49:06.887 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:49:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:49:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:49:07.261 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:49:07.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:49:07.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:49:07.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:49:07.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:49:07.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:49:07.287 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:07.293+0000 7fcb5ea0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:07.289 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:07.295+0000 7fcb5ea0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:07.290 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:07.296+0000 7fcb5ea0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:07.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:49:07.558 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:49:07.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:07.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:07.805 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:07.811+0000 7fcb5ea0e640 -1 Falling back to public interface 2024-03-30T11:49:07.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:08.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:08.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:08.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:49:08.843 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:49:08.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:08.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:09.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:10.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:10.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:10.140 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:49:10.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:49:10.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:10.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:10.147 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:10.153+0000 7fcb5ea0e640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:49:10.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:11.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:11.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:11.455 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:49:11.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:49:11.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:11.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:11.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:12.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:12.878 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:49:12.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:12.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:49:12.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:12.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:13.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:14.207 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:49:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:49:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:14.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:15.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:15.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:15.498 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:49:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:49:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:15.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:16.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:16.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:16.899 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:49:16.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:49:16.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:16.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:17.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:18.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:18.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:18.224 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:49:18.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:49:18.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:18.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:18.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:19.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:19.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:19.519 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:49:19.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:49:19.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:19.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:19.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:20.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:20.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:20.851 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:49:20.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:49:20.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:20.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:21.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:22.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:22.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:22.174 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:49:22.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:49:22.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:22.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:22.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:23.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:23.468 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:49:23.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:23.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:49:23.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:23.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:25.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:25.009 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:49:25.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:25.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:49:25.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:25.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:25.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:26.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:26.376 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:49:26.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:26.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:49:26.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:26.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:26.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:27.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:27.856 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:49:27.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:27.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:49:27.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:27.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:28.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:29.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:29.472 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:49:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:49:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:29.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:30.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:31.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:31.153 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:49:31.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:31.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:49:31.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:31.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:31.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:32.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:32.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:32.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:49:32.856 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:49:32.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:32.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:33.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:34.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:34.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:34.149 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:49:34.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:49:34.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:34.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:49:34.446 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1402367128,v1:127.0.0.1:6811/1402367128] [v2:127.0.0.1:6812/1402367128,v1:127.0.0.1:6813/1402367128] exists,up e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:49:34.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:49:34.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:49:34.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:49:34.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:49:34.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:49:34.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b57eca4-aec7-44fe-a436-a4f4ffba01da 2024-03-30T11:49:34.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2b57eca4-aec7-44fe-a436-a4f4ffba01da' 2024-03-30T11:49:34.454 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 2b57eca4-aec7-44fe-a436-a4f4ffba01da 2024-03-30T11:49:34.454 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:49:34.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBO/AdmeGlOHBAAUDXk722f5rSm1AqyPOHFtg== 2024-03-30T11:49:34.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBO/AdmeGlOHBAAUDXk722f5rSm1AqyPOHFtg=="}' 2024-03-30T11:49:34.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b57eca4-aec7-44fe-a436-a4f4ffba01da -i td/osd-scrub-repair/2/new.json 2024-03-30T11:49:34.840 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:49:34.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:49:34.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBO/AdmeGlOHBAAUDXk722f5rSm1AqyPOHFtg== --osd-uuid 2b57eca4-aec7-44fe-a436-a4f4ffba01da 2024-03-30T11:49:34.879 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.886+0000 7f615eb2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:34.916 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.923+0000 7f615eb2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:34.917 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.924+0000 7f615eb2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:34.917 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.924+0000 7f615eb2a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:49:34.917 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.924+0000 7f615eb2a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:49:34.918 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.924+0000 7f615eb2a640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:49:34.982 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:34.990+0000 7f615eb2a640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:49:40.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:49:40.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:49:40.965 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:49:40.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:49:40.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:49:41.330 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:49:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:49:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:49:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:49:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:49:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:49:41.356 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:41.363+0000 7f118cc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:41.358 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:41.365+0000 7f118cc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:41.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:41.366+0000 7f118cc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:49:41.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:49:41.615 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:49:41.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:41.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:41.874 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:41.881+0000 7f118cc0f640 -1 Falling back to public interface 2024-03-30T11:49:41.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:42.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:42.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:42.899 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:49:42.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:49:42.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:42.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:43.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:44.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:44.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:44.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:49:44.192 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:49:44.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:44.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:44.231 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:49:44.238+0000 7f118cc0f640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:49:44.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:45.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:45.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:45.548 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:49:45.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:49:45.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:45.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:45.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:46.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:46.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:46.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:49:46.883 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:49:46.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:46.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:47.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:48.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:48.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:48.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:49:48.279 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:49:48.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:48.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:48.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:49.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:49.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:49.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:49:49.624 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:49:49.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:49.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:49.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:50.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:50.917 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:49:50.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:50.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:49:50.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:50.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:51.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:52.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:52.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:52.258 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:49:52.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:49:52.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:52.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:52.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:53.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:53.549 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:49:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:49:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:53.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:54.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:54.886 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:49:54.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:54.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:49:54.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:54.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:55.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:56.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:56.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:56.254 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:49:56.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:49:56.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:56.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:56.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:57.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:57.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:57.548 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:49:57.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:49:57.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:57.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:58.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:49:59.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:49:59.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:49:59.394 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:49:59.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:49:59.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:49:59.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:49:59.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:00.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:00.920 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:50:00.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:00.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:50:00.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:00.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:50:01.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:02.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:02.304 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:50:02.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:02.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:50:02.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:02.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:50:03.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:04.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:04.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:04.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:50:04.066 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:50:04.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:04.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:50:04.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:05.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:05.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:05.544 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:50:05.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:50:05.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:05.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:50:05.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:06.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:50:06.936 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:50:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:06.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:50:07.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:08.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:08.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:08.398 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T11:50:08.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T11:50:08.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:08.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:50:08.689 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1317629737,v1:127.0.0.1:6819/1317629737] [v2:127.0.0.1:6820/1317629737,v1:127.0.0.1:6821/1317629737] exists,up 2b57eca4-aec7-44fe-a436-a4f4ffba01da 2024-03-30T11:50:08.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:50:08.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:50:08.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:50:08.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-30T11:50:08.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:50:08.989 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:50:09.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:50:09.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:50:09.373 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:50:09.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:50:10.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:50:12.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-30T11:50:12.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-30T11:50:12.541 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'foo' created 2024-03-30T11:50:12.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:50:13.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-03-30T11:50:13.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:50:13.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:50:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-30T11:50:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:50:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-30T11:50:13.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:50:13.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:50:14.759 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:50:14.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:50:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-30T11:50:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:50:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:50:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:50:15.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:50:15.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:50:15.775 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:50:15.775 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:50:15.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:50:15.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:50:15.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:50:15.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:50:15.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:50:15.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:50:15.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:50:15.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:50:15.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:50:16.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:50:16.155 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:16.155 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:50:16.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:50:16.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:16.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:50:16.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-30T11:50:16.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-30T11:50:16.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-30T11:50:16.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:16.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:50:16.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-30T11:50:16.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-30T11:50:16.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672973' 2024-03-30T11:50:16.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:16.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:50:16.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-30T11:50:16.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-30T11:50:16.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672973 2-64424509447' 2024-03-30T11:50:16.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:16.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-30T11:50:16.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:16.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:50:16.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-30T11:50:16.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:16.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-30T11:50:16.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-30T11:50:16.594 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836504 2024-03-30T11:50:16.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:16.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-30T11:50:16.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:16.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-30T11:50:16.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:16.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:50:16.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-30T11:50:16.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:16.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-30T11:50:16.878 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672973 2024-03-30T11:50:16.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-30T11:50:16.878 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:50:17.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-30T11:50:17.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:17.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-30T11:50:17.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:17.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:50:17.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-30T11:50:17.164 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:17.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-30T11:50:17.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-30T11:50:17.165 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509447 2024-03-30T11:50:17.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:50:17.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509447 2024-03-30T11:50:17.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:50:18.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:50:18.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:50:18.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509447 2024-03-30T11:50:18.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:50:18.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:18.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:19.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:50:19.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:50:19.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:50:19.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:50:19.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:50:19.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:50:19.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:50:19.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:50:19.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:50:19.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:50:19.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:19.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:19.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:50:19.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:50:19.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:50:19.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:50:19.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:50:19.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:50:19.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:50:19.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:50:20.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:50:20.161 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:20.162 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:50:20.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:50:20.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:20.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:50:20.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-30T11:50:20.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-30T11:50:20.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-30T11:50:20.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:20.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:50:20.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-30T11:50:20.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-30T11:50:20.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672975' 2024-03-30T11:50:20.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:20.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:50:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-30T11:50:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-30T11:50:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672975 2-64424509449' 2024-03-30T11:50:20.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:20.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-30T11:50:20.478 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:20.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:50:20.479 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-30T11:50:20.479 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:20.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-30T11:50:20.480 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836506 2024-03-30T11:50:20.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-30T11:50:20.481 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:20.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-30T11:50:20.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:20.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-30T11:50:20.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:20.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:50:20.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-30T11:50:20.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:20.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-30T11:50:20.800 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672975 2024-03-30T11:50:20.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-30T11:50:20.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:50:21.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-30T11:50:21.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:21.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-30T11:50:21.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:21.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:50:21.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-30T11:50:21.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:21.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-30T11:50:21.085 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509449 2024-03-30T11:50:21.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-30T11:50:21.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:50:21.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-30T11:50:21.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:50:21.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:21.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:21.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:50:21.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:50:21.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:50:21.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:50:21.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:50:21.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:50:21.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:50:21.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:50:21.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:50:21.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:50:21.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:21.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:22.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:50:22.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:50:22.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:50:22.347 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-30T11:50:22.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:50:22.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:50:22.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:50:22.822 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:50:22.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:50:23.232 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:50:23.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:50:23.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:50:23.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:50:23.461 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:50:23.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:50:23.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:50:24.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:50:24.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:50:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:50:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:50:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:50:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:50:26.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:50:26.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:50:26.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:50:26.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:50:26.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:50:26.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:50:26.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:50:26.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:50:26.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:50:26.552 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:26.559+0000 7f0ea1013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:26.554 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:26.561+0000 7f0ea1013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:26.555 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:26.562+0000 7f0ea1013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:26.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:50:26.820 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:50:26.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:26.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:27.070 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:27.077+0000 7f0ea1013640 -1 Falling back to public interface 2024-03-30T11:50:27.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:28.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:28.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:28.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:50:28.104 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:28.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:28.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:28.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:50:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:50:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:29.492 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:29.464+0000 7f0ea1013640 -1 osd.1 33 log_to_monitors true 2024-03-30T11:50:29.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:30.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:30.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:30.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:50:30.758 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:50:30.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:30.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [10,33) [v2:127.0.0.1:6810/2475619671,v1:127.0.0.1:6811/2475619671] [v2:127.0.0.1:6812/2475619671,v1:127.0.0.1:6813/2475619671] exists,up e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:50:31.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:50:31.175 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:50:31.175 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:50:31.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:50:31.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:50:31.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:50:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:50:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:50:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:50:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:50:31.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:50:31.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:50:31.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:50:31.547 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:31.547 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:50:31.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:50:31.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:31.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:50:31.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-30T11:50:31.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-30T11:50:31.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-30T11:50:31.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:31.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:50:31.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789954 2024-03-30T11:50:31.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789954 2024-03-30T11:50:31.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-158913789954' 2024-03-30T11:50:31.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:31.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:50:31.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-30T11:50:31.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-30T11:50:31.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-158913789954 2-64424509452' 2024-03-30T11:50:31.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:31.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-30T11:50:31.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:31.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:50:31.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-30T11:50:31.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:31.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-30T11:50:31.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-30T11:50:31.923 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836509 2024-03-30T11:50:31.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:32.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-30T11:50:32.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:50:33.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:50:33.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:33.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-30T11:50:33.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:33.506 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-158913789954 2024-03-30T11:50:33.506 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:33.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:50:33.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-158913789954 2024-03-30T11:50:33.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:33.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789954 2024-03-30T11:50:33.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 158913789954' 2024-03-30T11:50:33.509 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 158913789954 2024-03-30T11:50:33.509 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:50:33.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789954 -lt 158913789954 2024-03-30T11:50:33.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:33.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-30T11:50:33.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:33.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:50:33.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-30T11:50:33.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:33.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-30T11:50:33.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-30T11:50:33.797 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509452 2024-03-30T11:50:33.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:50:34.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-30T11:50:34.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:50:34.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:34.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:50:34.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:50:34.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:50:34.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:50:34.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:34.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-30T11:50:35.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:50:35.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:50:35.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:50:35.446 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:50:35.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:50:35.957 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:50:35.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:50:35.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:50:35.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:50:36.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-30T11:50:36.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-30T11:50:36.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:50:36.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-30T11:50:36.180 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-03-30T11:50:36.180 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-03-30T11:50:36.180 INFO:tasks.workunit.client.0.smithi003.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.0258e-05 s, 34.0 MB/s 2024-03-30T11:50:36.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:36.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:50:36.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:50:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:50:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:50:36.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:50:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:50:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:50:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:50:36.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:50:38.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:50:38.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:50:38.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:50:38.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:50:38.654 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:50:38.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:50:38.655 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:50:38.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:50:38.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:50:38.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:50:38.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:50:38.681 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:38.688+0000 7f603a410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:38.683 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:38.691+0000 7f603a410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:38.685 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:38.692+0000 7f603a410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:50:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:50:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:50:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:50:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:50:38.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:38.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:50:38.950 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:50:38.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:38.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:50:39.206 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:39.214+0000 7f603a410640 -1 Falling back to public interface 2024-03-30T11:50:39.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:40.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:40.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:40.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:50:40.238 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:40.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:40.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:50:40.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:41.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:50:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:50:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:41.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:50:41.657 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:41.664+0000 7f603a410640 -1 osd.0 41 log_to_monitors true 2024-03-30T11:50:41.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:42.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:42.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:42.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:50:42.842 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:50:42.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:42.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:50:43.303 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 46 up_thru 46 down_at 42 last_clean_interval [5,41) [v2:127.0.0.1:6802/2874791546,v1:127.0.0.1:6803/2874791546] [v2:127.0.0.1:6804/2874791546,v1:127.0.0.1:6805/2874791546] exists,up 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:50:43.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:50:43.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:50:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:50:43.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:50:43.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:50:43.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:50:43.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:50:43.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:50:43.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:50:43.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:50:43.728 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:43.729 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:50:43.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:50:43.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:43.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:50:43.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-03-30T11:50:43.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-03-30T11:50:43.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495618' 2024-03-30T11:50:43.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:43.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:50:43.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789957 2024-03-30T11:50:43.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789957 2024-03-30T11:50:43.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495618 1-158913789957' 2024-03-30T11:50:43.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:43.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:50:44.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-30T11:50:44.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-30T11:50:44.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495618 1-158913789957 2-64424509456' 2024-03-30T11:50:44.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:44.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-197568495618 2024-03-30T11:50:44.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:44.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:50:44.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-197568495618 2024-03-30T11:50:44.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:44.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-03-30T11:50:44.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 197568495618' 2024-03-30T11:50:44.055 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 197568495618 2024-03-30T11:50:44.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:44.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 197568495618 2024-03-30T11:50:44.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:50:45.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:50:45.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:45.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-03-30T11:50:45.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:45.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-158913789957 2024-03-30T11:50:45.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:45.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:50:45.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-158913789957 2024-03-30T11:50:45.642 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:45.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789957 2024-03-30T11:50:45.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 158913789957' 2024-03-30T11:50:45.643 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 158913789957 2024-03-30T11:50:45.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:50:45.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789957 -lt 158913789957 2024-03-30T11:50:45.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:45.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-30T11:50:45.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:45.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:50:45.927 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-30T11:50:45.927 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:45.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-30T11:50:45.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-30T11:50:45.929 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509456 2024-03-30T11:50:45.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:50:46.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-30T11:50:46.209 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:50:46.209 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:46.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:46.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:50:46.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:50:46.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:50:46.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:50:46.565 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:50:46.565 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:50:46.565 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:50:46.565 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:50:46.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:50:46.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:50:46.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:46.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:47.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:50:47.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:50:47.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:50:47.705 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:50:47.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:50:48.097 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:50:48.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:50:48.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:50:48.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:50:48.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-30T11:50:48.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:50:48.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:50:48.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-30T11:50:48.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:50:48.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:50:48.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:50:48.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:50:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:50:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-30T11:50:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:50:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:50:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:50:48.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:50:48.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:50:48.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-30T11:50:50.494 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:50:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:50:51.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:50:51.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:50:51.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:50:51.015 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:50:51.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:50:51.016 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:50:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:50:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:50:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:50:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:50:51.042 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:51.049+0000 7f9ebfeb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:51.044 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:51.051+0000 7f9ebfeb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:51.045 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:51.052+0000 7f9ebfeb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:50:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:50:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:50:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:50:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:50:51.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:50:51.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:51.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:50:51.311 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:50:51.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:51.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:51.566 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:51.574+0000 7f9ebfeb5640 -1 Falling back to public interface 2024-03-30T11:50:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:50:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:52.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:53.926 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:50:53.933+0000 7f9ebfeb5640 -1 osd.1 51 log_to_monitors true 2024-03-30T11:50:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:50:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:50:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:53.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:54.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:50:55.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:50:55.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:50:55.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:50:55.346 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:50:55.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:50:55.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:50:55.786 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 52 last_clean_interval [37,51) [v2:127.0.0.1:6810/1437691488,v1:127.0.0.1:6811/1437691488] [v2:127.0.0.1:6812/1437691488,v1:127.0.0.1:6813/1437691488] exists,up e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:50:55.787 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:50:55.788 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:50:55.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:50:55.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:50:55.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:50:55.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:50:55.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:50:55.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:50:55.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:50:55.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:50:55.868 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:50:56.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:50:56.181 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:50:56.181 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:50:56.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:50:56.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:56.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:50:56.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495621 2024-03-30T11:50:56.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495621 2024-03-30T11:50:56.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495621' 2024-03-30T11:50:56.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:56.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:50:56.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168578 2024-03-30T11:50:56.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168578 2024-03-30T11:50:56.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495621 1-240518168578' 2024-03-30T11:50:56.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:50:56.387 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:50:56.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-30T11:50:56.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-30T11:50:56.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495621 1-240518168578 2-64424509459' 2024-03-30T11:50:56.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:56.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-197568495621 2024-03-30T11:50:56.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:56.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:50:56.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-197568495621 2024-03-30T11:50:56.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:56.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495621 2024-03-30T11:50:56.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 197568495621' 2024-03-30T11:50:56.491 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 197568495621 2024-03-30T11:50:56.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:50:56.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495621 -lt 197568495621 2024-03-30T11:50:56.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:56.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168578 2024-03-30T11:50:56.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:56.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:50:56.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168578 2024-03-30T11:50:56.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:56.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168578 2024-03-30T11:50:56.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168578' 2024-03-30T11:50:56.804 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 240518168578 2024-03-30T11:50:56.804 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:50:57.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168578 -lt 240518168578 2024-03-30T11:50:57.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:50:57.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-30T11:50:57.144 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:50:57.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:50:57.146 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-30T11:50:57.146 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:50:57.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-30T11:50:57.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-30T11:50:57.147 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509459 2024-03-30T11:50:57.147 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:50:57.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-30T11:50:57.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:50:57.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:57.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:57.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:50:57.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:50:58.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:50:58.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:50:58.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:50:58.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:50:58.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:50:58.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:50:58.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:50:58.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:50:58.936 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:50:58.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:50:59.328 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:50:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:50:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:50:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:50:59.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-30T11:50:59.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-30T11:50:59.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:50:59.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-30T11:50:59.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:50:59.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:50:59.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:51:00.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:51:00.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:51:00.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:51:00.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:51:00.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:51:00.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:51:00.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:51:00.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:51:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:51:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:51:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:51:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:51:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:51:02.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:51:02.389 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:51:02.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:51:02.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:51:02.391 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:51:02.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:51:02.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:51:02.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:51:02.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:51:02.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:51:02.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:51:02.418 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:02.425+0000 7f7fe2a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:02.420 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:02.427+0000 7f7fe2a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:02.421 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:02.429+0000 7f7fe2a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:02.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:02.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:02.940 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:02.948+0000 7f7fe2a11640 -1 Falling back to public interface 2024-03-30T11:51:02.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:03.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:03.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:03.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:51:03.971 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:03.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:03.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:04.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:05.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:05.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:05.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:51:05.268 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:51:05.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:05.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:05.316 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:05.323+0000 7f7fe2a11640 -1 osd.0 61 log_to_monitors true 2024-03-30T11:51:05.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:06.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:06.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:06.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:51:06.563 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:51:06.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:06.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 62 last_clean_interval [46,61) [v2:127.0.0.1:6802/2107042776,v1:127.0.0.1:6803/2107042776] [v2:127.0.0.1:6804/2107042776,v1:127.0.0.1:6805/2107042776] exists,up 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:51:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:51:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:51:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:51:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:51:07.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:51:07.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:51:07.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:51:07.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:51:07.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:51:07.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:51:07.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:51:07.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:51:07.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:51:07.549 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:07.549 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:51:07.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:51:07.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:07.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:51:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-30T11:51:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-30T11:51:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-03-30T11:51:07.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:07.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:51:07.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168581 2024-03-30T11:51:07.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168581 2024-03-30T11:51:07.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-240518168581' 2024-03-30T11:51:07.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:07.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:51:07.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-30T11:51:07.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-30T11:51:07.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-240518168581 2-64424509463' 2024-03-30T11:51:07.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:07.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-03-30T11:51:07.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:07.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:51:07.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-03-30T11:51:07.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:07.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-30T11:51:07.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-03-30T11:51:07.877 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841538 2024-03-30T11:51:07.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:08.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-03-30T11:51:08.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:09.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:51:09.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:09.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-03-30T11:51:09.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:09.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168581 2024-03-30T11:51:09.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:09.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:51:09.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168581 2024-03-30T11:51:09.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:09.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168581 2024-03-30T11:51:09.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168581' 2024-03-30T11:51:09.453 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 240518168581 2024-03-30T11:51:09.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:51:09.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168581 -lt 240518168581 2024-03-30T11:51:09.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:09.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-30T11:51:09.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:09.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:51:09.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-30T11:51:09.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:09.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-30T11:51:09.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-30T11:51:09.738 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509463 2024-03-30T11:51:09.738 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:51:10.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-30T11:51:10.022 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:51:10.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:10.023 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:10.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:51:10.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:51:10.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:51:10.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:51:10.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:10.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:11.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:51:11.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:51:11.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:51:11.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:51:11.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:51:11.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:51:11.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:51:11.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:51:11.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:51:11.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:51:11.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:51:11.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:51:11.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:51:13.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:51:13.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:51:13.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:51:13.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:51:13.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:51:13.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:51:13.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:51:13.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.2 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:51:13.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:51:13.584 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:13.591+0000 7f8d22a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:13.586 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:13.593+0000 7f8d22a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:13.587 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:13.595+0000 7f8d22a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:13.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:14.108 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:14.116+0000 7f8d22a11640 -1 Falling back to public interface 2024-03-30T11:51:14.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:15.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:15.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:15.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:51:15.135 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:15.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:15.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:15.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:16.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:16.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:16.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:51:16.452 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:51:16.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:16.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:16.457 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:16.464+0000 7f8d22a11640 -1 osd.2 69 log_to_monitors true 2024-03-30T11:51:16.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:17.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:17.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:17.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:51:17.790 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:51:17.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:17.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/2760139651,v1:127.0.0.1:6819/2760139651] [v2:127.0.0.1:6820/2760139651,v1:127.0.0.1:6821/2760139651] exists,up 2b57eca4-aec7-44fe-a436-a4f4ffba01da 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:51:18.131 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:51:18.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:51:18.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:51:18.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:51:18.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:51:18.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:51:18.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:51:18.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:51:18.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:51:18.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:51:18.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:51:18.496 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:18.496 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:51:18.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:51:18.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:18.496 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:51:18.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-03-30T11:51:18.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-03-30T11:51:18.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-03-30T11:51:18.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:18.602 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:51:18.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168584 2024-03-30T11:51:18.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168584 2024-03-30T11:51:18.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-240518168584' 2024-03-30T11:51:18.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:18.707 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:51:18.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-30T11:51:18.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-30T11:51:18.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-240518168584 2-317827579906' 2024-03-30T11:51:18.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:18.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-03-30T11:51:18.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:18.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:51:18.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-03-30T11:51:18.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:18.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-03-30T11:51:18.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-03-30T11:51:18.815 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841541 2024-03-30T11:51:18.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:19.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-03-30T11:51:19.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:20.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:51:20.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:20.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-03-30T11:51:20.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:21.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:51:21.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:21.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-03-30T11:51:21.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:21.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168584 2024-03-30T11:51:21.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:21.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:51:21.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168584 2024-03-30T11:51:21.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:21.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168584 2024-03-30T11:51:21.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168584' 2024-03-30T11:51:21.713 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 240518168584 2024-03-30T11:51:21.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:51:21.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168585 -lt 240518168584 2024-03-30T11:51:21.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:21.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-30T11:51:21.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:21.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:51:21.997 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-30T11:51:21.997 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:21.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-30T11:51:21.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-30T11:51:21.998 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 317827579906 2024-03-30T11:51:21.998 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:51:22.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-30T11:51:22.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:51:22.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:22.279 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:22.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:51:22.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:51:22.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:51:22.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:51:22.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:51:22.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:51:22.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:51:22.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:51:22.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:51:22.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:51:22.907 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:22.907 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:23.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:51:23.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:51:23.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:51:23.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:51:23.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:51:23.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:51:23.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:51:23.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:51:23.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:51:23.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:51:23.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:51:23.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:51:23.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:51:23.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:51:23.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:51:25.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:51:25.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:51:25.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.2 2024-03-30T11:51:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:51:25.741 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:51:25.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:51:25.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:51:25.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:51:25.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:51:25.766 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:25.773+0000 7f23f9b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:25.768 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:25.775+0000 7f23f9b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:25.769 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:25.777+0000 7f23f9b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:26.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:26.284 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:26.291+0000 7f23f9b2a640 -1 Falling back to public interface 2024-03-30T11:51:26.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:27.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:27.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:27.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:51:27.313 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:27.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:27.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:27.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:28.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:28.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:28.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:51:28.609 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:51:28.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:28.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:28.660 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:28.666+0000 7f23f9b2a640 -1 osd.2 76 log_to_monitors true 2024-03-30T11:51:28.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:29.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:29.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:29.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:51:29.905 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:51:29.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:29.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:51:30.201 INFO:tasks.workunit.client.0.smithi003.stderr:osd.2 up in weight 1 up_from 81 up_thru 81 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/2061138119,v1:127.0.0.1:6819/2061138119] [v2:127.0.0.1:6820/2061138119,v1:127.0.0.1:6821/2061138119] exists,up 2b57eca4-aec7-44fe-a436-a4f4ffba01da 2024-03-30T11:51:30.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:51:30.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:51:30.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:51:30.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:51:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:51:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:51:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:51:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:51:30.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:51:30.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:51:30.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:51:30.639 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:30.639 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:51:30.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:51:30.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:30.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:51:30.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841545 2024-03-30T11:51:30.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841545 2024-03-30T11:51:30.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545' 2024-03-30T11:51:30.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:30.782 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:51:30.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168588 2024-03-30T11:51:30.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168588 2024-03-30T11:51:30.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-240518168588' 2024-03-30T11:51:30.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:30.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:51:30.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-30T11:51:30.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-30T11:51:30.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-240518168588 2-347892350978' 2024-03-30T11:51:30.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:30.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841545 2024-03-30T11:51:30.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:30.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:51:30.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841545 2024-03-30T11:51:30.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:30.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841545 2024-03-30T11:51:30.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841545' 2024-03-30T11:51:30.995 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841545 2024-03-30T11:51:30.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:31.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841545 2024-03-30T11:51:31.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:32.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:51:32.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:32.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841543 -lt 283467841545 2024-03-30T11:51:32.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:33.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:51:33.571 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841545 2024-03-30T11:51:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:33.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-240518168588 2024-03-30T11:51:33.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:33.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:51:33.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-240518168588 2024-03-30T11:51:33.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:33.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168588 2024-03-30T11:51:33.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 240518168588' 2024-03-30T11:51:33.867 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 240518168588 2024-03-30T11:51:33.868 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:51:34.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168588 -lt 240518168588 2024-03-30T11:51:34.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:34.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-03-30T11:51:34.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:34.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:51:34.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-03-30T11:51:34.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:34.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-30T11:51:34.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-03-30T11:51:34.198 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350978 2024-03-30T11:51:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:51:34.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-03-30T11:51:34.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:51:34.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:34.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:51:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:51:34.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:51:34.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:51:34.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:51:34.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:51:34.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:51:34.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:51:35.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:51:35.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:51:35.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:35.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:35.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:51:35.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:51:35.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:51:35.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-30T11:51:35.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:51:35.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-30T11:51:35.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-30T11:51:35.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:51:35.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:51:35.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-30T11:51:35.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:51:35.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:51:35.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:51:36.001 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:51:36.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:51:36.503 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:51:36.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:51:36.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:51:36.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:51:36.657 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-30T11:51:36.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:51:36.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:51:36.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-30T11:51:36.660 INFO:tasks.workunit.client.0.smithi003.stderr:2+0 records in 2024-03-30T11:51:36.660 INFO:tasks.workunit.client.0.smithi003.stderr:2+0 records out 2024-03-30T11:51:36.660 INFO:tasks.workunit.client.0.smithi003.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.5957e-05 s, 62.1 MB/s 2024-03-30T11:51:36.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:51:36.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:51:36.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:51:36.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:51:36.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:51:36.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:51:36.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:51:36.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:51:36.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:51:36.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:51:36.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:51:39.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:51:39.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:51:39.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:51:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:51:39.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:51:39.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:51:39.202 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:51:39.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:51:39.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:51:39.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:51:39.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:51:39.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:51:39.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:51:39.229 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:39.236+0000 7f7fb360f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:39.231 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:39.239+0000 7f7fb360f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:39.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:39.240+0000 7f7fb360f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:39.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:51:39.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:39.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:51:39.747 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:39.755+0000 7f7fb360f640 -1 Falling back to public interface 2024-03-30T11:51:39.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:40.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:40.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:40.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:51:40.811 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:40.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:40.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:51:41.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:42.085 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:42.092+0000 7f7fb360f640 -1 osd.1 85 log_to_monitors true 2024-03-30T11:51:42.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:42.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:42.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:51:42.105 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:51:42.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:42.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:51:42.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:43.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:43.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:43.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:51:43.412 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:51:43.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:43.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:51:43.786 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [56,85) [v2:127.0.0.1:6810/1146110966,v1:127.0.0.1:6811/1146110966] [v2:127.0.0.1:6812/1146110966,v1:127.0.0.1:6813/1146110966] exists,up e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:51:43.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:51:43.787 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:51:43.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:51:43.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:51:43.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:51:43.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:51:43.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:51:43.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:51:44.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:51:44.156 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:44.156 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:51:44.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:51:44.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:44.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:51:44.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841548 2024-03-30T11:51:44.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841548 2024-03-30T11:51:44.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548' 2024-03-30T11:51:44.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:44.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:51:44.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-03-30T11:51:44.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-03-30T11:51:44.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-386547056642' 2024-03-30T11:51:44.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:44.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:51:44.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350982 2024-03-30T11:51:44.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350982 2024-03-30T11:51:44.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-386547056642 2-347892350982' 2024-03-30T11:51:44.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:44.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841548 2024-03-30T11:51:44.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:44.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:51:44.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841548 2024-03-30T11:51:44.580 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841548 2024-03-30T11:51:44.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841548' 2024-03-30T11:51:44.582 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841548 2024-03-30T11:51:44.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:44.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841548 -lt 283467841548 2024-03-30T11:51:44.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:44.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-03-30T11:51:44.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:44.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:51:44.868 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-03-30T11:51:44.868 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:44.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-03-30T11:51:44.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-03-30T11:51:44.869 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 386547056642 2024-03-30T11:51:44.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:51:45.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-03-30T11:51:45.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:45.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350982 2024-03-30T11:51:45.152 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:45.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:51:45.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350982 2024-03-30T11:51:45.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:45.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350982 2024-03-30T11:51:45.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350982' 2024-03-30T11:51:45.154 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350982 2024-03-30T11:51:45.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:51:45.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350981 -lt 347892350982 2024-03-30T11:51:45.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:46.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:51:46.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:51:46.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350982 -lt 347892350982 2024-03-30T11:51:46.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:51:46.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:46.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:47.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:51:47.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:51:47.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:51:47.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:51:47.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:51:47.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:51:47.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:51:47.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:51:47.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:51:47.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:51:47.387 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:51:47.387 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:51:47.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:51:47.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:51:47.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:51:47.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:51:47.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:51:48.147 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:51:48.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:51:48.549 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:51:48.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:51:48.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:51:48.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:51:48.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-30T11:51:48.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:51:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:51:48.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:51:49.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:51:49.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:51:49.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:51:49.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:51:49.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:51:49.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:51:49.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:51:49.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:51:51.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:51:51.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:51:51.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:51:51.321 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:51:51.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:51:51.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:51:51.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:51:51.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:51:51.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:51:51.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:51:51.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:51:51.350 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:51.357+0000 7f8533147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:51.352 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:51.360+0000 7f8533147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:51.354 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:51.361+0000 7f8533147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:51:51.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:51.874 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:51.882+0000 7f8533147640 -1 Falling back to public interface 2024-03-30T11:51:51.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:52.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:52.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:52.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:51:52.897 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:52.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:52.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:53.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:54.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:54.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:54.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:51:54.191 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:51:54.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:54.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:54.256 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:51:54.263+0000 7f8533147640 -1 osd.0 94 log_to_monitors true 2024-03-30T11:51:54.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:51:55.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:51:55.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:51:55.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:51:55.590 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:51:55.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:51:55.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:51:55.997 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 99 up_thru 99 down_at 95 last_clean_interval [66,94) [v2:127.0.0.1:6802/2706149309,v1:127.0.0.1:6803/2706149309] [v2:127.0.0.1:6804/2706149309,v1:127.0.0.1:6805/2706149309] exists,up 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:51:55.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:51:55.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:51:55.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:51:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:51:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:51:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:51:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:51:55.998 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:51:55.999 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:51:55.999 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:51:55.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:51:55.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:51:55.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:51:56.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:51:56.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:51:56.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:51:56.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:51:56.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:51:56.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:51:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:51:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:51:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:51:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:51:56.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:56.452 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:51:56.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-30T11:51:56.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-30T11:51:56.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306' 2024-03-30T11:51:56.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:56.577 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:51:56.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056645 2024-03-30T11:51:56.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056645 2024-03-30T11:51:56.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056645' 2024-03-30T11:51:56.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:51:56.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:51:56.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350985 2024-03-30T11:51:56.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350985 2024-03-30T11:51:56.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762306 1-386547056645 2-347892350985' 2024-03-30T11:51:56.788 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:56.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762306 2024-03-30T11:51:56.788 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:56.789 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:51:56.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762306 2024-03-30T11:51:56.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:56.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-30T11:51:56.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762306' 2024-03-30T11:51:56.791 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 425201762306 2024-03-30T11:51:56.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:57.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 425201762306 2024-03-30T11:51:57.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:58.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:51:58.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:58.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 425201762306 2024-03-30T11:51:58.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:51:59.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:51:59.365 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:51:59.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-30T11:51:59.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:59.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056645 2024-03-30T11:51:59.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:59.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:51:59.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056645 2024-03-30T11:51:59.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:59.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056645 2024-03-30T11:51:59.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-03-30T11:51:59.667 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 386547056645 2024-03-30T11:51:59.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:51:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056646 -lt 386547056645 2024-03-30T11:51:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:51:59.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350985 2024-03-30T11:51:59.950 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:51:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:51:59.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350985 2024-03-30T11:51:59.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:51:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350985 2024-03-30T11:51:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-03-30T11:51:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350985 2024-03-30T11:51:59.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:52:00.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350985 -lt 347892350985 2024-03-30T11:52:00.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:52:00.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:00.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:00.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:52:00.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:52:00.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:52:00.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:52:00.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:52:00.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:52:00.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:52:00.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:52:00.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:52:00.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:52:00.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:00.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:01.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:52:01.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:52:01.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:52:01.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-30T11:52:01.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:52:01.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:52:01.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:52:01.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:52:01.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:52:01.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:52:01.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:52:01.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:52:01.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:52:01.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:52:03.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:52:03.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:52:03.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:52:03.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:52:03.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:52:03.633 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:52:03.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:52:03.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:52:03.635 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:52:03.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:52:03.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:52:03.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:52:03.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:52:03.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:52:03.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:52:03.661 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:03.668+0000 7fcea2ad9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:03.664 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:03.671+0000 7fcea2ad9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:03.665 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:03.672+0000 7fcea2ad9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:03.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:52:03.931 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:52:03.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:03.931 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:52:04.179 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:04.187+0000 7fcea2ad9640 -1 Falling back to public interface 2024-03-30T11:52:04.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:05.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:05.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:05.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:52:05.217 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:52:05.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:05.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:52:05.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:06.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:06.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:06.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:52:06.531 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:52:06.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:06.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:52:06.569 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:06.576+0000 7fcea2ad9640 -1 osd.1 101 log_to_monitors true 2024-03-30T11:52:07.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:08.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:08.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:08.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:52:08.003 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:52:08.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:08.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [90,101) [v2:127.0.0.1:6810/672003186,v1:127.0.0.1:6811/672003186] [v2:127.0.0.1:6812/672003186,v1:127.0.0.1:6813/672003186] exists,up e9aebfde-ac33-4166-baad-e5936e49e188 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:52:08.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:52:08.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:52:08.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:52:08.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:52:08.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:52:08.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:52:08.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:52:08.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:52:08.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:52:08.726 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:52:08.726 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:52:08.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:52:08.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:08.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:52:08.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-03-30T11:52:08.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-03-30T11:52:08.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309' 2024-03-30T11:52:08.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:08.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:52:08.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-03-30T11:52:08.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-03-30T11:52:08.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309 1-455266533378' 2024-03-30T11:52:08.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:08.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:52:09.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350989 2024-03-30T11:52:09.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350989 2024-03-30T11:52:09.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-425201762309 1-455266533378 2-347892350989' 2024-03-30T11:52:09.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:09.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-425201762309 2024-03-30T11:52:09.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:09.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:52:09.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-425201762309 2024-03-30T11:52:09.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:09.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-03-30T11:52:09.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 425201762309' 2024-03-30T11:52:09.082 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 425201762309 2024-03-30T11:52:09.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:09.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762308 -lt 425201762309 2024-03-30T11:52:09.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:52:10.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:52:10.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:10.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762310 -lt 425201762309 2024-03-30T11:52:10.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:10.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-03-30T11:52:10.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:10.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:52:10.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-03-30T11:52:10.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:10.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-03-30T11:52:10.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-03-30T11:52:10.664 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 455266533378 2024-03-30T11:52:10.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:52:10.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-03-30T11:52:10.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:10.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350989 2024-03-30T11:52:10.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:10.945 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:52:10.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350989 2024-03-30T11:52:10.946 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:10.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350989 2024-03-30T11:52:10.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350989' 2024-03-30T11:52:10.947 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350989 2024-03-30T11:52:10.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:52:11.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350989 -lt 347892350989 2024-03-30T11:52:11.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:52:11.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:11.233 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:11.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:52:11.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:52:11.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:52:11.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:52:11.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:52:11.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:52:11.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:52:11.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:52:11.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:52:11.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:52:11.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:11.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:52:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:52:12.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:52:12.713 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:52:12.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:52:13.106 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:52:13.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:52:13.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:52:13.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:52:13.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-30T11:52:13.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:52:13.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:52:13.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-30T11:52:13.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-30T11:52:13.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:52:13.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:52:13.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:52:13.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:52:13.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:52:13.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:52:13.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:52:13.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:52:13.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-30T11:52:13.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:52:13.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:52:13.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:52:13.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:52:13.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:52:13.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:52:15.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:52:15.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:52:15.722 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:52:15.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:52:15.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:52:15.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:52:15.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:52:15.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:52:15.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:52:15.749 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:15.756+0000 7fa603610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:15.752 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:15.759+0000 7fa603610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:15.753 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:15.760+0000 7fa603610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:16.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:52:16.018 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:52:16.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:16.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:16.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:16.275+0000 7fa603610640 -1 Falling back to public interface 2024-03-30T11:52:16.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:17.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:17.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:17.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:52:17.303 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:52:17.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:17.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:17.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:18.546 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:18.553+0000 7fa603610640 -1 osd.0 110 log_to_monitors true 2024-03-30T11:52:18.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:18.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:18.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:52:18.597 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:52:18.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:18.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:18.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:19.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:19.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:19.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:52:19.899 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:52:19.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:19.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [99,110) [v2:127.0.0.1:6802/2454880991,v1:127.0.0.1:6803/2454880991] [v2:127.0.0.1:6804/2454880991,v1:127.0.0.1:6805/2454880991] exists,up 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:52:20.298 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:52:20.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:52:20.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:52:20.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:52:20.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:52:20.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:52:20.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:52:20.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:52:20.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:52:20.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:52:20.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:52:20.666 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:52:20.666 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:52:20.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:52:20.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:20.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:52:20.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-30T11:52:20.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-30T11:52:20.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042' 2024-03-30T11:52:20.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:20.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:52:20.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-03-30T11:52:20.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-03-30T11:52:20.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381' 2024-03-30T11:52:20.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:20.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:52:20.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350992 2024-03-30T11:52:20.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350992 2024-03-30T11:52:20.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-493921239042 1-455266533381 2-347892350992' 2024-03-30T11:52:20.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:20.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-493921239042 2024-03-30T11:52:20.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:20.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:52:20.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-493921239042 2024-03-30T11:52:20.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:20.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-30T11:52:20.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-03-30T11:52:20.993 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 493921239042 2024-03-30T11:52:20.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:21.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 493921239042 2024-03-30T11:52:21.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:52:22.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:52:22.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:22.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 493921239042 2024-03-30T11:52:22.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:52:23.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:52:23.575 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:23.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239042 -lt 493921239042 2024-03-30T11:52:23.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:23.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533381 2024-03-30T11:52:23.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:23.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:52:23.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533381 2024-03-30T11:52:23.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:23.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-03-30T11:52:23.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533381' 2024-03-30T11:52:23.873 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 455266533381 2024-03-30T11:52:23.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:52:24.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533382 -lt 455266533381 2024-03-30T11:52:24.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:24.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350992 2024-03-30T11:52:24.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:24.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:52:24.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350992 2024-03-30T11:52:24.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:24.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350992 2024-03-30T11:52:24.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350992' 2024-03-30T11:52:24.162 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350992 2024-03-30T11:52:24.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:52:24.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350992 -lt 347892350992 2024-03-30T11:52:24.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:52:24.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:24.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:24.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:52:24.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:52:24.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:52:24.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:52:24.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:52:24.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:52:24.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:52:24.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:52:25.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:52:25.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:52:25.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:25.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:52:25.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:52:25.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:52:25.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:52:28.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:52:28.211 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:52:28.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:52:28.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:52:28.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:52:28.213 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:52:28.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:52:28.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:52:28.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:52:28.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:52:28.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:52:28.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:52:28.240 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:28.247+0000 7f4a030c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:28.242 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:28.249+0000 7f4a030c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:28.243 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:28.250+0000 7f4a030c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:28.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:28.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:28.760 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:28.768+0000 7f4a030c8640 -1 Falling back to public interface 2024-03-30T11:52:28.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:29.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:29.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:29.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:52:29.820 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:52:29.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:29.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:30.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:31.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:31.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:31.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:52:31.112 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:52:31.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:31.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:31.144 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:31.151+0000 7f4a030c8640 -1 osd.0 117 log_to_monitors true 2024-03-30T11:52:31.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:32.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:32.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:32.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:52:32.524 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:52:32.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:32.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/3315318415,v1:127.0.0.1:6803/3315318415] [v2:127.0.0.1:6804/3315318415,v1:127.0.0.1:6805/3315318415] exists,up 02eb325d-a9fd-4236-b3c5-fb6b95edd7a9 2024-03-30T11:52:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:52:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:52:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:52:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:52:32.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:52:32.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:52:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:52:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:52:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:52:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:52:33.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:52:33.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:52:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:52:33.285 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:52:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:52:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:52:33.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:33.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:52:33.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-03-30T11:52:33.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-03-30T11:52:33.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-03-30T11:52:33.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:33.391 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:52:33.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533385 2024-03-30T11:52:33.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533385 2024-03-30T11:52:33.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533385' 2024-03-30T11:52:33.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:52:33.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:52:33.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350995 2024-03-30T11:52:33.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350995 2024-03-30T11:52:33.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-455266533385 2-347892350995' 2024-03-30T11:52:33.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:33.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-03-30T11:52:33.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:33.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:52:33.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-03-30T11:52:33.635 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:33.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-03-30T11:52:33.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-03-30T11:52:33.636 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 523986010114 2024-03-30T11:52:33.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:33.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-03-30T11:52:33.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:52:34.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:52:34.920 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:52:35.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-03-30T11:52:35.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:35.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533385 2024-03-30T11:52:35.214 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:35.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:52:35.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533385 2024-03-30T11:52:35.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:35.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533385 2024-03-30T11:52:35.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533385' 2024-03-30T11:52:35.217 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 455266533385 2024-03-30T11:52:35.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:52:35.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533385 2024-03-30T11:52:35.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:52:35.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350995 2024-03-30T11:52:35.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:52:35.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:52:35.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350995 2024-03-30T11:52:35.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:52:35.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350995 2024-03-30T11:52:35.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350995' 2024-03-30T11:52:35.523 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350995 2024-03-30T11:52:35.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:52:35.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350996 -lt 347892350995 2024-03-30T11:52:35.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:52:35.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:35.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:36.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:52:36.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:52:36.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:52:36.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:52:36.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:52:36.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:52:36.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:52:36.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:52:36.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:52:36.440 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:52:36.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:52:36.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:52:36.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:52:36.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:52:36.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:52:36.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-30T11:52:36.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-30T11:52:36.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:52:36.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-30T11:52:36.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-30T11:52:36.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:37.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:52:37.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:37.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-30T11:52:37.641 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-30T11:52:37.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:37.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T11:52:37.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:37.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:52:37.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:37.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:38.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:38.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:38.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:52:38.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:38.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:52:38.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:38.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:52:39.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:39.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:40.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:40.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:40.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:52:40.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:40.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:52:40.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:40.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:52:40.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:40.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:41.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:41.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:41.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:52:41.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:41.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:52:41.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:41.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:52:41.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:41.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:42.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:42.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:42.813 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:52:42.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:42.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:52:42.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:42.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:52:43.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:52:38.448095+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:43.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:52:43.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-30T11:52:43.127 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-30T11:52:43.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-30T11:52:43.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T11:52:43.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-30T11:52:43.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-30T11:52:43.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-30T11:52:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-03-30T11:52:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:52:43.161 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:52:43.161 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:52:43.161 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:52:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:52:43.185 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T11:52:43.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:52:43.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:52:43.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:52:43.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:52:43.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-30T11:52:43.209 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:43.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:52:43.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:43.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-30T11:52:43.753 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-30T11:50:14.640762+0000 last_deep_scrub_stamp 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:43.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:52:44.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:44.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:45.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:45.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:45.044 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:52:45.044 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:45.044 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:52:45.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:45.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:52:45.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:45.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:46.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:46.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:46.335 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:52:46.335 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:46.335 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:52:46.335 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:46.336 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:52:46.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:46.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:47.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:47.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:47.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:52:47.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:47.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:52:47.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:47.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:52:47.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:50:14.640762+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:47.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:52:48.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:52:48.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:52:48.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:52:48.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:52:48.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:52:48.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:52:48.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:52:49.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:52:44.627428+0000 '>' 2024-03-30T11:50:14.640762+0000 2024-03-30T11:52:49.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:52:49.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-30T11:52:49.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-30T11:52:49.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-30T11:52:49.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T11:52:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-30T11:52:49.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-30T11:52:49.263 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:52:49.266 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:52:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:52:49.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:52:49.290 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:52:49.290 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:52:49.290 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:52:49.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:52:49.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:52:49.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:52:49.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-30T11:52:49.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-30T11:52:49.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-30T11:52:49.698 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' removed 2024-03-30T11:52:49.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:52:49.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:52:49.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:52:49.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:52:49.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:52:49.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:52:49.820 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:52:49.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:52:49.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:52:49.822 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:52:49.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:52:49.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:52:49.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:52:49.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:52:49.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:52:49.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:52:49.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:52:49.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:52:49.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:52:49.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:52:49.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:52:49.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:52:49.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:52:49.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:52:49.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:52:49.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:52:49.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:52:49.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:52:49.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:52:49.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:52:49.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:52:49.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:52:49.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:52:49.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:52:49.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:52:49.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:52:49.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:52:49.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:49.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:49.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:52:49.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:52:49.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:52:49.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:52:49.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:52:49.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:49.863 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:49.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-03-30T11:52:49.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:52:49.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T11:52:50.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:50.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T11:52:50.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:52:50.433 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:52:50.434 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:52:50.434 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:50.434 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:50.434 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:52:50.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:52:50.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:52:50.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:52:50.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:52:50.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:52:50.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-30T11:52:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:52:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:52:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:52:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:52:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:52:50.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:51.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:52:51.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:52:51.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-30T11:52:51.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-30T11:52:51.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-30T11:52:51.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:52:51.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:52:51.444 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:52:51.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:52:51.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:52:51.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:52:51.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:52:51.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:52:51.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:52:51.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:52:51.447 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:52:51.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5c1f6c66-00cc-441c-90bd-546e64ffbcb8' 2024-03-30T11:52:51.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:52:51.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAT/QdmU4rWGxAAQCmNENyMi6A5ARZgtZ839g== 2024-03-30T11:52:51.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAT/QdmU4rWGxAAQCmNENyMi6A5ARZgtZ839g=="}' 2024-03-30T11:52:51.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:52:51.664 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:52:51.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:52:51.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAT/QdmU4rWGxAAQCmNENyMi6A5ARZgtZ839g== --osd-uuid 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:52:51.703 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.710+0000 7f7843260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:51.715 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.722+0000 7f7843260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:51.716 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.724+0000 7f7843260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:51.717 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.724+0000 7f7843260640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:52:51.717 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.724+0000 7f7843260640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:52:51.718 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.724+0000 7f7843260640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:52:51.807 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:51.814+0000 7f7843260640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:52:57.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:52:57.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:52:57.600 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:52:57.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:52:57.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:52:57.971 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:52:57.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:52:57.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:52:57.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:52:57.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:52:57.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:52:57.997 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:58.004+0000 7fb6f9413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:57.999 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:58.006+0000 7fb6f9413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:58.000 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:58.007+0000 7fb6f9413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:52:58.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:52:58.253 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:52:58.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:58.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:58.513 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:52:58.521+0000 7fb6f9413640 -1 Falling back to public interface 2024-03-30T11:52:58.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:52:59.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:52:59.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:52:59.535 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:52:59.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:52:59.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:52:59.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:52:59.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:00.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:00.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:00.830 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:53:00.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:53:00.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:00.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:00.838 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:00.845+0000 7fb6f9413640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:53:01.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:02.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:02.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:02.273 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:53:02.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:53:02.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:02.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:02.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:03.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:03.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:03.580 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:53:03.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:53:03.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:03.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:03.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:04.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:04.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:04.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:53:04.874 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:53:04.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:04.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:05.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:06.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:06.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:06.165 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:53:06.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:53:06.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:06.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:06.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:07.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:07.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:07.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:53:07.461 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:53:07.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:07.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:07.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:08.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:08.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:08.774 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:53:08.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:53:08.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:08.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:09.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:10.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:10.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:10.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:53:10.071 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:53:10.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:10.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:10.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:11.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:11.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:11.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:53:11.386 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:53:11.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:11.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:11.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:12.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:12.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:12.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:53:12.780 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:53:12.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:12.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:13.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:14.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:14.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:14.375 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:53:14.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:53:14.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:14.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:15.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:16.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:16.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:16.026 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:53:16.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:53:16.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:16.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:16.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:17.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:17.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:17.714 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:53:17.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:53:17.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:17.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:18.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:19.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:19.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:19.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:53:19.622 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:53:19.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:19.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:20.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:21.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:21.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:21.155 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:53:21.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:53:21.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:21.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:21.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:22.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:22.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:22.643 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:53:22.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:53:22.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:22.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:53:22.997 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1819558589,v1:127.0.0.1:6803/1819558589] [v2:127.0.0.1:6804/1819558589,v1:127.0.0.1:6805/1819558589] exists,up 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:53:23.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:53:23.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:53:23.002 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:53:23.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:53:23.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 05834c1a-b4aa-4ba2-ab98-c92b34091826' 2024-03-30T11:53:23.005 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:53:23.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:53:23.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAz/QdmUBCNARAA0xGipl3DxDR+vxG3X3HsFg== 2024-03-30T11:53:23.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAz/QdmUBCNARAA0xGipl3DxDR+vxG3X3HsFg=="}' 2024-03-30T11:53:23.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05834c1a-b4aa-4ba2-ab98-c92b34091826 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:53:23.403 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:53:23.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:53:23.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAz/QdmUBCNARAA0xGipl3DxDR+vxG3X3HsFg== --osd-uuid 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:53:23.445 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.452+0000 7fb8f4e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:23.488 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.495+0000 7fb8f4e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:23.490 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.496+0000 7fb8f4e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:23.490 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.497+0000 7fb8f4e0f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:53:23.490 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.497+0000 7fb8f4e0f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:53:23.490 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.497+0000 7fb8f4e0f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:53:23.563 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:23.570+0000 7fb8f4e0f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:53:29.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:53:29.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:53:29.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:53:29.562 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:53:29.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:53:29.938 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:53:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:53:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:53:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:53:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:53:29.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:53:29.964 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:29.971+0000 7fbfd34d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:29.966 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:29.973+0000 7fbfd34d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:29.967 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:29.974+0000 7fbfd34d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:30.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:53:30.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:53:30.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:53:30.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:53:30.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:53:30.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:30.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:53:30.226 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:53:30.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:30.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:30.481 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:30.489+0000 7fbfd34d9640 -1 Falling back to public interface 2024-03-30T11:53:30.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:31.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:31.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:31.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:53:31.535 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:53:31.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:31.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:31.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:32.828 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:53:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:32.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:53:32.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:32.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:32.923 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:32.930+0000 7fbfd34d9640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:53:33.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:34.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:34.140 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:53:34.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:34.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:53:34.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:34.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:34.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:35.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:35.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:35.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:53:35.435 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:53:35.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:35.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:35.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:36.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:36.791 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:53:36.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:36.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:53:36.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:36.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:37.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:38.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:38.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:38.085 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:53:38.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:53:38.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:38.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:38.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:39.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:39.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:39.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:53:39.379 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:53:39.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:39.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:39.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:40.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:40.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:40.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:53:40.735 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:53:40.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:40.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:41.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:42.030 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:53:42.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:42.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:53:42.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:42.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:42.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:43.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:43.327 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:53:43.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:43.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:53:43.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:43.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:43.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:44.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:44.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:44.621 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:53:44.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:53:44.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:44.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:45.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:46.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:46.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:46.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:53:46.219 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:53:46.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:46.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:46.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:47.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:47.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:47.595 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:53:47.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:53:47.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:47.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:47.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:48.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:48.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:48.999 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:53:49.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:53:49.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:49.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:49.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:50.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:50.716 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:53:50.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:50.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:53:50.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:50.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:51.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:52.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:52.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:52.322 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:53:52.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:53:52.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:52.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:52.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:53.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:53.921 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:53:53.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:53.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:53:53.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:53.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:54.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:53:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:53:55.707 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:53:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:53:55.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:53:55.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:53:55.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:53:56.010 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/584976994,v1:127.0.0.1:6811/584976994] [v2:127.0.0.1:6812/584976994,v1:127.0.0.1:6813/584976994] exists,up 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:53:56.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:53:56.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:53:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=09b8c7b2-c469-4e95-b4fc-33b9facaedeb 2024-03-30T11:53:56.017 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 09b8c7b2-c469-4e95-b4fc-33b9facaedeb 2024-03-30T11:53:56.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 09b8c7b2-c469-4e95-b4fc-33b9facaedeb' 2024-03-30T11:53:56.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:53:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBU/QdmDnRRAhAA/myj1Pmm0okgbkrwM8hJQQ== 2024-03-30T11:53:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBU/QdmDnRRAhAA/myj1Pmm0okgbkrwM8hJQQ=="}' 2024-03-30T11:53:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 09b8c7b2-c469-4e95-b4fc-33b9facaedeb -i td/osd-scrub-repair/2/new.json 2024-03-30T11:53:56.365 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:53:56.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T11:53:56.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBU/QdmDnRRAhAA/myj1Pmm0okgbkrwM8hJQQ== --osd-uuid 09b8c7b2-c469-4e95-b4fc-33b9facaedeb 2024-03-30T11:53:56.405 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.411+0000 7f8d06e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:56.441 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.448+0000 7f8d06e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.449+0000 7f8d06e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:53:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.449+0000 7f8d06e25640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:53:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.449+0000 7f8d06e25640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:53:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.449+0000 7f8d06e25640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T11:53:56.508 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:53:56.515+0000 7f8d06e25640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T11:54:02.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T11:54:02.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:54:02.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T11:54:02.499 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T11:54:02.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:54:02.875 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T11:54:02.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T11:54:02.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:54:02.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:54:02.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:54:02.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:54:02.901 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:02.908+0000 7f693a4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:02.903 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:02.910+0000 7f693a4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:02.904 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:02.911+0000 7f693a4d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:03.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:54:03.162 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:54:03.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:03.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:03.418 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:03.426+0000 7f693a4d9640 -1 Falling back to public interface 2024-03-30T11:54:03.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:04.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:04.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:04.443 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:54:04.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:54:04.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:04.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:04.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:05.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:05.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:05.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:54:05.735 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:54:05.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:05.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:05.863 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:05.870+0000 7f693a4d9640 -1 osd.2 0 log_to_monitors true 2024-03-30T11:54:06.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:07.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:07.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:07.065 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:54:07.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:54:07.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:07.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:07.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:08.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:08.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:08.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:54:08.359 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:54:08.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:08.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:08.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:09.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:09.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:09.653 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:54:09.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:54:09.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:09.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:09.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:10.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:10.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:10.999 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:54:11.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:54:11.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:11.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:11.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:12.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:12.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:12.299 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:54:12.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:54:12.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:12.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:12.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:13.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:13.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:13.642 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:54:13.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:54:13.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:13.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:13.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:14.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:14.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:14.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:54:14.980 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:54:14.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:14.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:15.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:16.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:16.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:16.277 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:54:16.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:54:16.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:16.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:16.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:17.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:17.657 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:54:17.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:17.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:54:17.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:17.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:18.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:19.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:19.585 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:54:19.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:19.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:54:19.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:19.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:20.027 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:21.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:21.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:21.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:54:21.029 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:54:21.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:21.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:21.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:22.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:22.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:22.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:54:22.657 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:54:22.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:22.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:23.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:24.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:24.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:24.410 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:54:24.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:54:24.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:24.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:25.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:26.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:26.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:26.043 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:54:26.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:54:26.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:26.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:26.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:27.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:27.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:27.699 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:54:27.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:54:27.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:27.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:27.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:28.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:28.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:28.999 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:54:29.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:54:29.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:29.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:54:29.292 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3315498766,v1:127.0.0.1:6819/3315498766] [v2:127.0.0.1:6820/3315498766,v1:127.0.0.1:6821/3315498766] exists,up 09b8c7b2-c469-4e95-b4fc-33b9facaedeb 2024-03-30T11:54:29.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:54:29.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:54:29.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:54:29.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-30T11:54:29.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:54:29.596 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:54:29.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:54:29.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:54:29.985 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:54:29.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:54:30.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:54:32.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-30T11:54:32.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-30T11:54:33.194 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'foo' created 2024-03-30T11:54:33.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:54:34.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-03-30T11:54:34.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T11:54:34.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T11:54:34.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-30T11:54:34.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T11:54:34.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-30T11:54:34.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T11:54:34.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T11:54:35.348 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T11:54:35.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:54:36.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-30T11:54:36.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-30T11:54:36.795 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 3 allow_ec_overwrites to true 2024-03-30T11:54:36.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T11:54:36.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:54:36.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:54:36.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:54:36.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:54:36.808 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:54:36.808 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:54:36.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:54:36.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:54:36.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:54:36.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:54:36.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:54:36.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:54:36.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:54:36.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:54:36.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:54:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:54:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:54:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:54:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:54:37.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:37.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:54:37.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-30T11:54:37.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-30T11:54:37.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-30T11:54:37.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:37.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:54:37.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-30T11:54:37.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-30T11:54:37.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-03-30T11:54:37.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:37.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:54:37.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-30T11:54:37.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-30T11:54:37.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509451' 2024-03-30T11:54:37.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:37.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-30T11:54:37.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:37.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:54:37.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-30T11:54:37.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:37.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-30T11:54:37.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-30T11:54:37.594 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836503 2024-03-30T11:54:37.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:54:37.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-30T11:54:37.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:37.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-30T11:54:37.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:37.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:54:37.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-30T11:54:37.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:37.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-30T11:54:37.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-30T11:54:37.914 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672974 2024-03-30T11:54:37.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:54:38.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-30T11:54:38.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:38.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-30T11:54:38.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:38.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:54:38.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-30T11:54:38.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:38.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-30T11:54:38.199 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509451 2024-03-30T11:54:38.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-30T11:54:38.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:54:38.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-30T11:54:38.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:54:38.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:54:38.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:54:38.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:54:39.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:54:39.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:54:39.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:54:39.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:54:39.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:54:39.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:54:39.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:54:39.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:54:39.507 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:54:39.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:54:39.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:54:39.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:54:39.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:54:39.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:54:39.593 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:54:39.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:54:39.890 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:54:39.890 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:54:39.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:54:39.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:39.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:54:39.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-30T11:54:39.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-30T11:54:39.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-30T11:54:39.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:39.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:54:40.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-30T11:54:40.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-30T11:54:40.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672975' 2024-03-30T11:54:40.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:40.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:54:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-30T11:54:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-30T11:54:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672975 2-64424509453' 2024-03-30T11:54:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:40.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-30T11:54:40.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:40.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:54:40.207 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-30T11:54:40.207 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:40.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-30T11:54:40.208 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836505 2024-03-30T11:54:40.208 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-30T11:54:40.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:54:40.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-30T11:54:40.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:54:41.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:54:41.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:54:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-30T11:54:41.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:41.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-30T11:54:41.851 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:54:41.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-30T11:54:41.853 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-30T11:54:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-30T11:54:41.854 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672975 2024-03-30T11:54:41.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:54:42.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-03-30T11:54:42.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:42.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-30T11:54:42.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:42.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:54:42.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-30T11:54:42.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:42.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-30T11:54:42.182 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509453 2024-03-30T11:54:42.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-30T11:54:42.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:54:42.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-30T11:54:42.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:54:42.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:54:42.466 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:54:42.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:54:42.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:54:43.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:54:43.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:54:43.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:54:43.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:54:43.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:54:43.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:54:43.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:54:43.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-30T11:54:43.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:54:43.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-30T11:54:43.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-30T11:54:43.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:54:43.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:54:43.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-30T11:54:43.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:54:43.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:54:43.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:54:43.945 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:54:43.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:54:44.339 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:54:44.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:54:44.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:54:44.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:54:44.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:54:44.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:54:44.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:54:45.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:54:45.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:47.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:54:47.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:54:47.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:54:47.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:54:47.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:54:47.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:54:47.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:54:47.730 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:47.737+0000 7f62ff00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:47.732 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:47.739+0000 7f62ff00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:47.733 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:47.740+0000 7f62ff00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:54:47.994 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:54:47.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:47.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:54:48.247 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:48.255+0000 7f62ff00d640 -1 Falling back to public interface 2024-03-30T11:54:48.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:49.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:49.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:49.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:54:49.277 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:54:49.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:49.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:54:49.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:50.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:50.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:50.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:54:50.569 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:54:50.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:50.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:54:50.677 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:50.684+0000 7f62ff00d640 -1 osd.1 35 log_to_monitors true 2024-03-30T11:54:50.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:54:51.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:54:51.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:54:51.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:54:51.876 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:54:51.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:54:51.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:54:52.220 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/2780783068,v1:127.0.0.1:6811/2780783068] [v2:127.0.0.1:6812/2780783068,v1:127.0.0.1:6813/2780783068] exists,up 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:54:52.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:54:52.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:54:52.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:54:52.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:54:52.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:54:52.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:54:52.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:54:52.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:54:52.624 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:54:52.624 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:54:52.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:54:52.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:52.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:54:52.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-03-30T11:54:52.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-03-30T11:54:52.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-03-30T11:54:52.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:52.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:54:52.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-30T11:54:52.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-30T11:54:52.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-171798691842' 2024-03-30T11:54:52.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:54:52.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:54:52.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-30T11:54:52.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-30T11:54:52.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-171798691842 2-64424509456' 2024-03-30T11:54:52.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:52.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-03-30T11:54:52.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:52.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:54:52.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-03-30T11:54:52.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:52.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-03-30T11:54:52.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-03-30T11:54:52.988 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836508 2024-03-30T11:54:52.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:54:53.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-03-30T11:54:53.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:54:54.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:54:54.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:54:54.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836508 2024-03-30T11:54:54.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:54.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-30T11:54:54.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:54.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:54:54.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-30T11:54:54.612 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:54.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-30T11:54:54.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-30T11:54:54.614 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 171798691842 2024-03-30T11:54:54.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:54:54.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-30T11:54:54.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:54:54.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-30T11:54:54.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:54:54.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:54:54.898 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-30T11:54:54.898 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:54:54.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-30T11:54:54.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-30T11:54:54.899 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509456 2024-03-30T11:54:54.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:54:55.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-30T11:54:55.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:54:55.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:54:55.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:54:55.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:54:55.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:54:55.540 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:54:55.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:54:55.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:54:55.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:54:55.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:54:55.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:54:55.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:54:55.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:54:55.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:54:55.822 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:54:56.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:54:56.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:54:56.558 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:54:56.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:54:57.052 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:54:57.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:54:57.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:54:57.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:54:57.283 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-30T11:54:57.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-30T11:54:57.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:54:57.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-30T11:54:57.286 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-03-30T11:54:57.286 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-03-30T11:54:57.286 INFO:tasks.workunit.client.0.smithi003.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.1797e-05 s, 33.1 MB/s 2024-03-30T11:54:57.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:54:57.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:54:57.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:54:59.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:54:59.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:54:59.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:54:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:54:59.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:54:59.980 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:59.987+0000 7fcd573bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:59.982 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:59.989+0000 7fcd573bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:54:59.983 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:54:59.990+0000 7fcd573bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:00.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:55:00.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:55:00.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:55:00.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:55:00.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:55:00.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:00.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:55:00.245 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:55:00.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:00.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:00.501 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:00.508+0000 7fcd573bd640 -1 Falling back to public interface 2024-03-30T11:55:00.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:01.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:01.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:01.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:55:01.554 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:01.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:01.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:01.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:55:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:55:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:02.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:02.919 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:02.926+0000 7fcd573bd640 -1 osd.0 43 log_to_monitors true 2024-03-30T11:55:03.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:04.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:04.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:04.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:55:04.149 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:55:04.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:04.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/1814957180,v1:127.0.0.1:6803/1814957180] [v2:127.0.0.1:6804/1814957180,v1:127.0.0.1:6805/1814957180] exists,up 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:55:04.519 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:55:04.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:55:04.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:55:04.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:55:04.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:55:04.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:55:04.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:55:04.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:55:04.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:55:04.605 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:55:04.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:55:04.993 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:04.993 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:55:04.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:55:04.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:04.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:55:05.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-03-30T11:55:05.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-03-30T11:55:05.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-03-30T11:55:05.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:05.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:55:05.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-30T11:55:05.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-30T11:55:05.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691845' 2024-03-30T11:55:05.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:05.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:55:05.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-30T11:55:05.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-30T11:55:05.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691845 2-64424509460' 2024-03-30T11:55:05.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:05.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-03-30T11:55:05.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:05.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:55:05.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-03-30T11:55:05.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:05.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-03-30T11:55:05.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-03-30T11:55:05.311 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 206158430210 2024-03-30T11:55:05.311 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:05.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-03-30T11:55:05.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:55:06.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:55:06.592 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:06.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-03-30T11:55:06.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:06.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-03-30T11:55:06.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:06.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:55:06.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-03-30T11:55:06.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-30T11:55:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-03-30T11:55:06.887 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 171798691845 2024-03-30T11:55:06.887 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:55:07.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-30T11:55:07.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:07.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-30T11:55:07.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:07.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:55:07.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-30T11:55:07.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:07.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-30T11:55:07.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-30T11:55:07.217 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509460 2024-03-30T11:55:07.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:55:07.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-30T11:55:07.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:55:07.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:07.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:07.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:55:07.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:07.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:07.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:07.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:07.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:07.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:07.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:08.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:55:08.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:08.117 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:08.117 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:08.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:55:08.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:55:08.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:55:08.838 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:55:08.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:55:09.357 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:55:09.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:55:09.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:55:09.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:55:09.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:55:09.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:55:09.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:55:10.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:55:10.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-30T11:55:10.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:55:10.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:55:10.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:55:10.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:55:10.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:55:10.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-30T11:55:12.104 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:55:12.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:55:12.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:55:12.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:55:12.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:55:12.652 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:12.658+0000 7fefec4b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:12.654 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:12.661+0000 7fefec4b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:12.655 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:12.662+0000 7fefec4b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:12.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:55:12.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:55:12.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:55:12.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:55:12.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:55:12.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:12.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:55:12.921 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:55:12.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:12.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:55:13.167 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:13.174+0000 7fefec4b5640 -1 Falling back to public interface 2024-03-30T11:55:13.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:14.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:55:14.203 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:14.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:14.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:55:14.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:15.479 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:15.486+0000 7fefec4b5640 -1 osd.1 52 log_to_monitors true 2024-03-30T11:55:15.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:15.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:55:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:55:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:15.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:55:15.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:16.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:16.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:16.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:55:16.857 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:55:16.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:16.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [40,52) [v2:127.0.0.1:6810/2168190591,v1:127.0.0.1:6811/2168190591] [v2:127.0.0.1:6812/2168190591,v1:127.0.0.1:6813/2168190591] exists,up 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:55:17.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:55:17.274 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:55:17.274 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:55:17.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:55:17.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:55:17.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:55:17.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:55:17.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:55:17.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:55:17.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:55:17.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:55:17.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:55:17.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:55:17.640 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:17.641 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:55:17.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:55:17.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:17.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:55:17.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-03-30T11:55:17.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-03-30T11:55:17.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213' 2024-03-30T11:55:17.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:17.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:55:17.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-30T11:55:17.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-30T11:55:17.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874' 2024-03-30T11:55:17.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:17.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:55:17.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-30T11:55:17.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-30T11:55:17.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874 2-64424509463' 2024-03-30T11:55:17.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:17.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430213 2024-03-30T11:55:17.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:17.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:55:17.975 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430213 2024-03-30T11:55:17.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:17.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-03-30T11:55:17.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430213' 2024-03-30T11:55:17.977 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 206158430213 2024-03-30T11:55:17.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:18.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430213 2024-03-30T11:55:18.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:55:19.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:55:19.261 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:19.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430212 -lt 206158430213 2024-03-30T11:55:19.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:55:20.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T11:55:20.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:20.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430214 -lt 206158430213 2024-03-30T11:55:20.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:20.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-30T11:55:20.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:20.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:55:20.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-30T11:55:20.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:20.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-30T11:55:20.851 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-30T11:55:20.851 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 244813135874 2024-03-30T11:55:20.852 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:55:21.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-30T11:55:21.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:21.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-30T11:55:21.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:21.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:55:21.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-30T11:55:21.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:21.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-30T11:55:21.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-30T11:55:21.136 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509463 2024-03-30T11:55:21.136 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:55:21.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509463 2024-03-30T11:55:21.418 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:55:21.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:21.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:21.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:21.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:22.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:55:22.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:22.054 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:22.055 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:22.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:55:22.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:55:22.901 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:55:22.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:55:23.302 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:55:23.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:55:23.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:55:23.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:55:23.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-30T11:55:23.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-30T11:55:23.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:55:23.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-30T11:55:23.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-30T11:55:24.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-30T11:55:24.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:55:24.019 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:55:24.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:55:24.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:55:24.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:55:24.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:55:24.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-30T11:55:26.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:55:26.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:55:26.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:55:26.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:55:26.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:55:26.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:55:26.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:55:26.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:55:26.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:55:26.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:55:26.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:55:26.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:55:26.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:55:26.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:55:26.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:55:26.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:55:26.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:55:26.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:55:26.523 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:55:26.524 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:55:26.524 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:55:26.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:26.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:26.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:55:26.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:55:26.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:55:26.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:55:26.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:55:26.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:55:26.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:55:26.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:55:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:55:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:55:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:55:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:55:26.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:55:26.526 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:55:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:55:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:55:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:55:26.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:55:26.546 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:26.553+0000 7f8a0c4ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:26.548 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:26.555+0000 7f8a0c4ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:26.549 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:26.557+0000 7f8a0c4ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:27.062 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:27.070+0000 7f8a0c4ce640 -1 Falling back to public interface 2024-03-30T11:55:27.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:55:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:28.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:28.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:29.351 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:29.358+0000 7f8a0c4ce640 -1 osd.0 61 log_to_monitors true 2024-03-30T11:55:29.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:29.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:29.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:55:29.456 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:55:29.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:29.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:55:29.791 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 66 up_thru 57 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/599695917,v1:127.0.0.1:6803/599695917] [v2:127.0.0.1:6804/599695917,v1:127.0.0.1:6805/599695917] exists,up 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:55:29.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:55:29.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:55:29.793 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:55:29.793 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:55:29.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:55:29.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:55:29.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:55:29.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:55:29.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:55:29.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:55:29.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:55:29.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:55:29.879 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:55:30.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:55:30.163 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:30.163 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:55:30.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:55:30.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:30.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:55:30.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-30T11:55:30.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-30T11:55:30.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-03-30T11:55:30.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:30.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:55:30.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135877 2024-03-30T11:55:30.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135877 2024-03-30T11:55:30.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135877' 2024-03-30T11:55:30.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:30.648 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:55:30.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-03-30T11:55:30.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-03-30T11:55:30.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-244813135877 2-64424509467' 2024-03-30T11:55:30.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:30.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-03-30T11:55:30.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:30.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:55:30.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-03-30T11:55:30.792 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:30.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-30T11:55:30.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-03-30T11:55:30.793 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841538 2024-03-30T11:55:30.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:31.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-03-30T11:55:31.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:55:32.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:55:32.194 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:32.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-03-30T11:55:32.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:32.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135877 2024-03-30T11:55:32.490 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:32.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:55:32.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135877 2024-03-30T11:55:32.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:32.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135877 2024-03-30T11:55:32.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135877' 2024-03-30T11:55:32.493 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 244813135877 2024-03-30T11:55:32.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:55:32.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135877 2024-03-30T11:55:32.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:32.825 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-03-30T11:55:32.825 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:32.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:55:32.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-03-30T11:55:32.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-03-30T11:55:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-03-30T11:55:32.828 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 64424509467 2024-03-30T11:55:32.828 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:55:33.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-03-30T11:55:33.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:55:33.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:33.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:33.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:33.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:55:33.746 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:33.747 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:33.747 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:34.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-03-30T11:55:34.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-30T11:55:34.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:55:34.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-30T11:55:34.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:55:34.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:55:34.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:55:34.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:34.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:34.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-30T11:55:34.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:34.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:34.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:34.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-03-30T11:55:34.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:55:34.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:55:35.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:55:35.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:55:35.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:55:35.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:55:35.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:55:35.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:55:35.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:35.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:35.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:35.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:35.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:35.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:35.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:35.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:55:35.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:35.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:35.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T11:55:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 0 2024-03-30T11:55:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:55:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T11:55:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:55:36.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:55:36.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:55:36.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:36.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:36.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:36.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:36.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:36.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:36.120 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:36.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:55:36.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:36.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:36.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:55:36.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:55:37.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:55:37.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:55:37.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:55:37.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:55:37.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:55:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:55:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:37.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:37.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:37.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:37.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:37.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:37.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:37.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:55:37.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:37.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:37.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:37.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T11:55:38.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:55:38.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:55:38.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:55:38.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:55:38.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:55:38.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:55:38.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:55:38.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:55:38.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:55:38.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:55:38.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T11:55:38.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:55:38.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:38.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:39.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:55:39.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:39.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:39.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:55:39.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:55:39.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:55:39.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:55:39.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-30T11:55:41.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:55:41.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:55:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:55:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:55:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:55:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:55:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.2 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:55:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:55:41.883 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:41.890+0000 7fd111a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:41.885 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:41.892+0000 7fd111a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:41.886 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:41.893+0000 7fd111a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:55:42.151 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:55:42.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:42.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:42.401 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:42.409+0000 7fd111a0f640 -1 Falling back to public interface 2024-03-30T11:55:42.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:43.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:43.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:43.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:55:43.455 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:43.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:43.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:43.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:44.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:44.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:44.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:55:44.750 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:55:44.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:44.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:44.844 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:44.852+0000 7fd111a0f640 -1 osd.2 69 log_to_monitors true 2024-03-30T11:55:45.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:46.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:46.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:46.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:55:46.045 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:55:46.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:46.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/1447185983,v1:127.0.0.1:6819/1447185983] [v2:127.0.0.1:6820/1447185983,v1:127.0.0.1:6821/1447185983] exists,up 09b8c7b2-c469-4e95-b4fc-33b9facaedeb 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:55:46.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:55:46.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:55:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:55:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:55:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:55:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:55:46.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:55:46.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:55:46.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:55:46.754 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:46.754 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:55:46.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:55:46.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:46.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:55:46.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841542 2024-03-30T11:55:46.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841542 2024-03-30T11:55:46.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841542' 2024-03-30T11:55:46.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:46.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:55:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135882 2024-03-30T11:55:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135882 2024-03-30T11:55:46.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841542 1-244813135882' 2024-03-30T11:55:46.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:46.965 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:55:47.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-30T11:55:47.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-30T11:55:47.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841542 1-244813135882 2-317827579906' 2024-03-30T11:55:47.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:47.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841542 2024-03-30T11:55:47.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:47.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:55:47.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841542 2024-03-30T11:55:47.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:47.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841542 2024-03-30T11:55:47.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841542' 2024-03-30T11:55:47.085 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841542 2024-03-30T11:55:47.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:47.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841542 2024-03-30T11:55:47.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:55:48.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:55:48.367 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:48.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841542 -lt 283467841542 2024-03-30T11:55:48.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:48.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135882 2024-03-30T11:55:48.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:48.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:55:48.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135882 2024-03-30T11:55:48.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:48.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135882 2024-03-30T11:55:48.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135882' 2024-03-30T11:55:48.665 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 244813135882 2024-03-30T11:55:48.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:55:48.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135882 -lt 244813135882 2024-03-30T11:55:48.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:48.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-30T11:55:48.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:48.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:55:48.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-30T11:55:48.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:48.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-30T11:55:48.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-30T11:55:48.962 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 317827579906 2024-03-30T11:55:48.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:55:49.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-30T11:55:49.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:55:49.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:49.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:49.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:55:49.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:49.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:49.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:49.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:49.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:49.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:49.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:49.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:55:49.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:49.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:49.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:55:50.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:55:50.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:55:50.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:55:52.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:55:52.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:52.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:55:52.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:55:52.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T11:55:52.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T11:55:52.666 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.2 2024-03-30T11:55:52.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:55:52.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T11:55:52.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T11:55:52.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:55:52.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:55:52.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:55:52.693 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:52.700+0000 7f79569fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:52.696 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:52.703+0000 7f79569fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:52.697 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:52.704+0000 7f79569fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:55:52.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T11:55:52.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:52.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:53.212 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:53.219+0000 7f79569fd640 -1 Falling back to public interface 2024-03-30T11:55:53.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:54.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:54.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:54.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:55:54.244 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:54.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:54.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:54.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:55:55.507 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:55:55.514+0000 7f79569fd640 -1 osd.2 76 log_to_monitors true 2024-03-30T11:55:55.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:55:55.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:55:55.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:55:55.560 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:55:55.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:55:55.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:osd.2 up in weight 1 up_from 81 up_thru 74 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/2871000277,v1:127.0.0.1:6819/2871000277] [v2:127.0.0.1:6820/2871000277,v1:127.0.0.1:6821/2871000277] exists,up 09b8c7b2-c469-4e95-b4fc-33b9facaedeb 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:55:55.930 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:55:55.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:55:55.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:55:55.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:55:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:55:56.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:55:56.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:55:56.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:55:56.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:55:56.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:55:56.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:55:56.368 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:55:56.368 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:55:56.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:55:56.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:56.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:55:56.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841545 2024-03-30T11:55:56.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841545 2024-03-30T11:55:56.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545' 2024-03-30T11:55:56.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:56.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:55:56.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135885 2024-03-30T11:55:56.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135885 2024-03-30T11:55:56.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-244813135885' 2024-03-30T11:55:56.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:55:56.689 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:55:56.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-30T11:55:56.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-30T11:55:56.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-244813135885 2-347892350978' 2024-03-30T11:55:56.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:56.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841545 2024-03-30T11:55:56.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:56.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:55:56.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841545 2024-03-30T11:55:56.945 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:56.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841545 2024-03-30T11:55:56.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841545' 2024-03-30T11:55:56.947 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841545 2024-03-30T11:55:56.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:57.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841544 -lt 283467841545 2024-03-30T11:55:57.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:55:58.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:55:58.332 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:55:58.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841545 2024-03-30T11:55:58.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:58.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135885 2024-03-30T11:55:58.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:58.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:55:58.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135885 2024-03-30T11:55:58.629 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:58.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135885 2024-03-30T11:55:58.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135885' 2024-03-30T11:55:58.630 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 244813135885 2024-03-30T11:55:58.630 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:55:58.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135885 -lt 244813135885 2024-03-30T11:55:58.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:55:58.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-03-30T11:55:58.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:55:58.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:55:58.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-03-30T11:55:58.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:55:58.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-30T11:55:58.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-03-30T11:55:58.915 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350978 2024-03-30T11:55:58.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:55:59.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350978 -lt 347892350978 2024-03-30T11:55:59.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:55:59.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:59.199 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:55:59.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:55:59.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:55:59.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:55:59.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:55:59.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:55:59.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:00.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T11:56:00.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T11:56:00.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T11:56:00.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T11:56:00.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:56:00.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T11:56:00.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:56:00.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:00.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:00.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:00.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:00.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:00.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:00.287 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:56:00.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T11:56:00.568 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:56:00.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:00.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T11:56:00.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T11:56:01.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T11:56:01.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T11:56:01.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T11:56:01.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T11:56:01.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:01.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:56:01.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:56:01.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:56:01.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:01.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:56:02.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:56:02.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:56:02.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:56:02.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:56:02.510 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:56:02.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:56:03.071 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:56:03.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:56:03.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:56:03.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:56:03.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-30T11:56:03.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:56:03.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:56:03.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-30T11:56:03.311 INFO:tasks.workunit.client.0.smithi003.stderr:2+0 records in 2024-03-30T11:56:03.311 INFO:tasks.workunit.client.0.smithi003.stderr:2+0 records out 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.917e-05 s, 59.2 MB/s 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:56:03.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:56:03.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:56:03.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:56:03.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:56:03.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:56:03.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:56:03.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:56:03.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:56:03.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:56:03.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:56:03.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:56:03.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:56:05.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:56:05.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:56:05.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:56:05.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:56:05.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:56:05.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:56:05.782 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:56:05.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:56:05.783 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:56:05.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:56:05.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:56:05.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:56:05.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:56:05.809 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:05.816+0000 7fd4f5262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:05.828 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:05.836+0000 7fd4f5262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:05.829 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:05.837+0000 7fd4f5262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:06.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:06.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:06.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:06.360 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:06.368+0000 7fd4f5262640 -1 Falling back to public interface 2024-03-30T11:56:07.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:07.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:07.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:56:07.355 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:07.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:07.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:07.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:08.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:08.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:08.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:56:08.701 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:56:08.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:08.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:08.846 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:08.853+0000 7fd4f5262640 -1 osd.1 86 log_to_monitors true 2024-03-30T11:56:08.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:09.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:56:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:56:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:10.292 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [57,86) [v2:127.0.0.1:6810/3479145884,v1:127.0.0.1:6811/3479145884] [v2:127.0.0.1:6812/3479145884,v1:127.0.0.1:6813/3479145884] exists,up 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:56:10.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:56:10.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:56:10.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:56:10.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:56:10.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:56:10.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:56:10.380 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:56:10.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:56:10.757 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:10.757 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:56:10.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:56:10.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:10.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:56:10.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841549 2024-03-30T11:56:10.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841549 2024-03-30T11:56:10.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841549' 2024-03-30T11:56:10.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:10.871 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:56:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-03-30T11:56:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-03-30T11:56:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841549 1-390842023938' 2024-03-30T11:56:11.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:11.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:56:11.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350982 2024-03-30T11:56:11.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350982 2024-03-30T11:56:11.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841549 1-390842023938 2-347892350982' 2024-03-30T11:56:11.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:11.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841549 2024-03-30T11:56:11.153 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:11.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:56:11.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841549 2024-03-30T11:56:11.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:11.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841549 2024-03-30T11:56:11.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841549' 2024-03-30T11:56:11.156 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 283467841549 2024-03-30T11:56:11.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:11.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841548 -lt 283467841549 2024-03-30T11:56:11.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:56:12.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:56:12.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841549 -lt 283467841549 2024-03-30T11:56:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:12.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023938 2024-03-30T11:56:12.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:12.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:56:12.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023938 2024-03-30T11:56:12.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-03-30T11:56:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-03-30T11:56:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 390842023938 2024-03-30T11:56:12.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:56:13.082 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023938 -lt 390842023938 2024-03-30T11:56:13.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:13.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350982 2024-03-30T11:56:13.083 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:13.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:56:13.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350982 2024-03-30T11:56:13.085 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:13.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350982 2024-03-30T11:56:13.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350982' 2024-03-30T11:56:13.086 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350982 2024-03-30T11:56:13.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:56:13.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350982 -lt 347892350982 2024-03-30T11:56:13.414 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:56:13.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:13.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:13.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:56:13.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:13.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:13.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:13.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:13.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:13.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:13.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:56:14.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:56:14.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:56:14.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:14.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:14.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:56:14.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:56:14.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:56:14.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:56:14.891 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:56:14.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:56:15.284 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:56:15.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:56:15.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:56:15.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:56:15.539 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-30T11:56:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-30T11:56:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:56:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:56:15.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:56:15.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:56:15.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:56:15.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:56:15.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:56:15.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:56:15.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:56:15.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:56:15.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:56:18.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:56:18.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:56:18.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:56:18.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:56:18.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:56:18.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:56:18.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:56:18.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:56:18.045 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:18.051+0000 7f6cfd20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:18.047 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:18.053+0000 7f6cfd20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:18.048 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:18.054+0000 7f6cfd20d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:18.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:18.565 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:18.572+0000 7f6cfd20d640 -1 Falling back to public interface 2024-03-30T11:56:18.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:19.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:19.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:19.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:56:19.604 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:19.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:19.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:19.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:20.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:20.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:20.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:56:20.899 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:56:20.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:20.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:21.042 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:21.048+0000 7f6cfd20d640 -1 osd.0 95 log_to_monitors true 2024-03-30T11:56:21.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:22.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:22.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:22.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:56:22.197 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:56:22.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:22.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [66,95) [v2:127.0.0.1:6802/598883320,v1:127.0.0.1:6803/598883320] [v2:127.0.0.1:6804/598883320,v1:127.0.0.1:6805/598883320] exists,up 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:56:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:56:22.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:56:22.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:56:22.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:56:22.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:56:22.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:56:22.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:56:22.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:56:23.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:56:23.029 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:23.029 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:56:23.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:56:23.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:23.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:56:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-30T11:56:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-30T11:56:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602' 2024-03-30T11:56:23.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:23.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:56:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023941 2024-03-30T11:56:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023941 2024-03-30T11:56:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023941' 2024-03-30T11:56:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:23.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:56:23.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350985 2024-03-30T11:56:23.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350985 2024-03-30T11:56:23.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729602 1-390842023941 2-347892350985' 2024-03-30T11:56:23.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:23.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729602 2024-03-30T11:56:23.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:23.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:56:23.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729602 2024-03-30T11:56:23.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:23.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-30T11:56:23.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-03-30T11:56:23.377 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 429496729602 2024-03-30T11:56:23.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:23.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 429496729602 2024-03-30T11:56:23.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:56:24.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:56:24.662 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:24.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729602 -lt 429496729602 2024-03-30T11:56:24.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:24.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023941 2024-03-30T11:56:24.955 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:24.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:56:24.956 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023941 2024-03-30T11:56:24.956 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023941 2024-03-30T11:56:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023941' 2024-03-30T11:56:24.958 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 390842023941 2024-03-30T11:56:24.958 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:56:25.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023941 -lt 390842023941 2024-03-30T11:56:25.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:25.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350985 2024-03-30T11:56:25.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:25.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:56:25.240 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350985 2024-03-30T11:56:25.241 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:25.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350985 2024-03-30T11:56:25.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350985' 2024-03-30T11:56:25.242 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350985 2024-03-30T11:56:25.242 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:56:25.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350985 -lt 347892350985 2024-03-30T11:56:25.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:56:25.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:25.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:25.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:56:25.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:25.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:25.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:25.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:25.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:25.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:25.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:56:26.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:56:26.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:56:26.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:26.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:56:26.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:56:26.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:56:26.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:56:26.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:56:26.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:56:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:56:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:56:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:56:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:56:26.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:56:28.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:28.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:56:28.941 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:56:28.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:56:28.942 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:56:28.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:56:28.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:56:28.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:56:28.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:56:28.969 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:28.976+0000 7f5b8aa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:28.971 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:28.979+0000 7f5b8aa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:28.972 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:28.980+0000 7f5b8aa11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:29.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:29.486 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:29.494+0000 7f5b8aa11640 -1 Falling back to public interface 2024-03-30T11:56:29.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:30.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:30.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:30.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:56:30.518 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:30.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:30.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:30.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:31.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:31.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:31.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:56:31.817 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:56:31.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:31.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:31.944 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:31.952+0000 7f5b8aa11640 -1 osd.1 102 log_to_monitors true 2024-03-30T11:56:32.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:33.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:33.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:33.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:56:33.225 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:56:33.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:33.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/158075036,v1:127.0.0.1:6811/158075036] [v2:127.0.0.1:6812/158075036,v1:127.0.0.1:6813/158075036] exists,up 05834c1a-b4aa-4ba2-ab98-c92b34091826 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:56:33.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:56:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:56:33.595 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:56:33.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:56:33.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:56:33.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:56:33.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:56:33.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:56:33.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:56:33.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:56:33.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:56:33.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:56:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:56:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:56:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:56:34.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:34.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:56:34.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729605 2024-03-30T11:56:34.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729605 2024-03-30T11:56:34.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605' 2024-03-30T11:56:34.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:34.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:56:34.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-03-30T11:56:34.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-03-30T11:56:34.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-459561500674' 2024-03-30T11:56:34.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:34.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:56:34.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350988 2024-03-30T11:56:34.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350988 2024-03-30T11:56:34.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-429496729605 1-459561500674 2-347892350988' 2024-03-30T11:56:34.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:34.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-429496729605 2024-03-30T11:56:34.325 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:34.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:56:34.326 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-429496729605 2024-03-30T11:56:34.326 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:34.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729605 2024-03-30T11:56:34.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 429496729605' 2024-03-30T11:56:34.327 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 429496729605 2024-03-30T11:56:34.328 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:34.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729605 2024-03-30T11:56:34.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:56:35.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:56:35.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:35.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729605 -lt 429496729605 2024-03-30T11:56:35.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:35.971 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500674 2024-03-30T11:56:35.971 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:35.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:56:35.973 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500674 2024-03-30T11:56:35.973 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:35.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-03-30T11:56:35.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-03-30T11:56:35.974 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 459561500674 2024-03-30T11:56:35.974 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:56:36.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500674 -lt 459561500674 2024-03-30T11:56:36.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:36.257 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350988 2024-03-30T11:56:36.257 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:36.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:56:36.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350988 2024-03-30T11:56:36.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:36.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350988 2024-03-30T11:56:36.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-03-30T11:56:36.260 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350988 2024-03-30T11:56:36.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:56:36.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350988 -lt 347892350988 2024-03-30T11:56:36.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:56:36.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:36.554 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:36.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:36.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:56:37.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:56:37.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:56:37.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:37.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:37.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:56:37.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:56:38.012 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:56:38.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:56:38.404 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:56:38.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:56:38.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:56:38.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:56:38.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-30T11:56:38.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-30T11:56:38.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-30T11:56:38.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-30T11:56:38.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-30T11:56:38.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:56:38.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:56:38.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:56:38.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:56:38.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:56:38.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:56:38.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:56:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:56:40.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:56:40.933 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:56:40.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:56:40.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:56:40.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:56:40.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:56:40.935 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:56:40.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:56:40.935 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:56:40.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:56:40.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:56:40.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:56:40.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:56:40.961 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:40.968+0000 7fb813baa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:40.963 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:40.970+0000 7fb813baa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:40.964 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:40.971+0000 7fb813baa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:41.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:41.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:41.485 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:41.493+0000 7fb813baa640 -1 Falling back to public interface 2024-03-30T11:56:41.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:42.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:42.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:42.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:56:42.510 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:42.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:42.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:42.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:43.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:43.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:43.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:56:43.815 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:56:43.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:43.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:44.069 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:44.073+0000 7fb813baa640 -1 osd.0 111 log_to_monitors true 2024-03-30T11:56:44.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:45.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:45.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:45.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:56:45.187 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:56:45.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:45.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 116 up_thru 116 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/3264809833,v1:127.0.0.1:6803/3264809833] [v2:127.0.0.1:6804/3264809833,v1:127.0.0.1:6805/3264809833] exists,up 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:56:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:56:45.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:56:45.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:56:45.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:56:45.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:56:45.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:56:45.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:56:45.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:56:45.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:56:46.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:56:46.005 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:46.005 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:56:46.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:56:46.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:46.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:56:46.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-30T11:56:46.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-30T11:56:46.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338' 2024-03-30T11:56:46.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:46.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:56:46.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500677 2024-03-30T11:56:46.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500677 2024-03-30T11:56:46.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-459561500677' 2024-03-30T11:56:46.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:46.216 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:56:46.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350992 2024-03-30T11:56:46.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350992 2024-03-30T11:56:46.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-459561500677 2-347892350992' 2024-03-30T11:56:46.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:46.320 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-498216206338 2024-03-30T11:56:46.321 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:46.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:56:46.322 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-498216206338 2024-03-30T11:56:46.322 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:46.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-30T11:56:46.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-03-30T11:56:46.323 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 498216206338 2024-03-30T11:56:46.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:46.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 498216206338 2024-03-30T11:56:46.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:56:47.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:56:47.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:47.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-03-30T11:56:47.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:47.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500677 2024-03-30T11:56:47.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:47.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:56:47.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500677 2024-03-30T11:56:47.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:47.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500677 2024-03-30T11:56:47.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500677' 2024-03-30T11:56:47.914 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 459561500677 2024-03-30T11:56:47.914 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:56:48.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500678 -lt 459561500677 2024-03-30T11:56:48.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:48.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350992 2024-03-30T11:56:48.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:48.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:56:48.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350992 2024-03-30T11:56:48.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:48.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350992 2024-03-30T11:56:48.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350992' 2024-03-30T11:56:48.205 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350992 2024-03-30T11:56:48.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:56:48.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350992 -lt 347892350992 2024-03-30T11:56:48.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:56:48.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:48.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:48.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:56:48.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:48.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:48.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:48.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:48.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:48.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:48.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:56:49.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:56:49.141 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:56:49.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:49.142 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:56:49.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:56:49.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:56:49.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-30T11:56:51.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:56:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:56:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:56:51.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:56:51.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:56:51.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:56:51.966 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:51.973+0000 7f62cea8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:51.968 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:51.976+0000 7f62cea8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:51.970 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:51.977+0000 7f62cea8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:56:52.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:56:52.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:56:52.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:56:52.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:56:52.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:56:52.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:52.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:56:52.232 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:56:52.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:52.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:52.495 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:52.503+0000 7f62cea8b640 -1 Falling back to public interface 2024-03-30T11:56:52.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:53.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:53.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:53.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:56:53.518 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:53.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:53.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:53.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:54.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:54.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:54.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:56:54.811 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:56:54.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:54.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:54.849 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:56:54.857+0000 7f62cea8b640 -1 osd.0 118 log_to_monitors true 2024-03-30T11:56:55.228 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:56:56.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:56:56.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:56:56.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:56:56.230 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:56:56.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:56:56.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:56:56.562 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/1152094847,v1:127.0.0.1:6803/1152094847] [v2:127.0.0.1:6804/1152094847,v1:127.0.0.1:6805/1152094847] exists,up 5c1f6c66-00cc-441c-90bd-546e64ffbcb8 2024-03-30T11:56:56.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:56:56.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:56:56.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:56:56.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:56:56.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:56:56.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:56:56.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:56:56.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:56:56.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:56:56.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:56:56.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:56:56.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:56:56.947 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:56:56.947 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T11:56:56.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:56:56.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:56.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:56:57.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-03-30T11:56:57.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-03-30T11:56:57.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410' 2024-03-30T11:56:57.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:57.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:56:57.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500680 2024-03-30T11:56:57.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500680 2024-03-30T11:56:57.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-459561500680' 2024-03-30T11:56:57.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:56:57.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T11:56:57.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350995 2024-03-30T11:56:57.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350995 2024-03-30T11:56:57.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-459561500680 2-347892350995' 2024-03-30T11:56:57.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:57.263 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-528280977410 2024-03-30T11:56:57.263 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:57.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:56:57.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-528280977410 2024-03-30T11:56:57.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:57.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-03-30T11:56:57.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-03-30T11:56:57.266 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 528280977410 2024-03-30T11:56:57.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:57.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 528280977410 2024-03-30T11:56:57.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:56:58.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:56:58.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:56:58.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-03-30T11:56:58.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:58.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500680 2024-03-30T11:56:58.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:58.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:56:58.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500680 2024-03-30T11:56:58.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:58.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500680 2024-03-30T11:56:58.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500680' 2024-03-30T11:56:58.845 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 459561500680 2024-03-30T11:56:58.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:56:59.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500681 -lt 459561500680 2024-03-30T11:56:59.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:56:59.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350995 2024-03-30T11:56:59.166 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:56:59.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T11:56:59.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350995 2024-03-30T11:56:59.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:56:59.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350995 2024-03-30T11:56:59.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350995' 2024-03-30T11:56:59.169 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.2 seq 347892350995 2024-03-30T11:56:59.169 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T11:56:59.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350995 -lt 347892350995 2024-03-30T11:56:59.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:56:59.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:56:59.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:56:59.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:56:59.802 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:57:00.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:57:00.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:57:00.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:57:00.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:57:00.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:57:00.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:57:00.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:57:00.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-30T11:57:00.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-30T11:57:00.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T11:57:00.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-30T11:57:00.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-30T11:57:00.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:57:00.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:00.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:57:01.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:01.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-30T11:57:01.272 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:01.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:57:01.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:54:35.305443+0000 '>' 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:01.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:57:02.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:57:02.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:57:02.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:57:02.567 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:02.567 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:57:02.567 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:02.567 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:57:02.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:54:35.305443+0000 '>' 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:02.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:57:03.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:57:03.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:57:03.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T11:57:03.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:03.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T11:57:03.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:03.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T11:57:04.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:57:01.680620+0000 '>' 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:04.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:57:04.152 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-30T11:57:04.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-30T11:57:04.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-30T11:57:04.181 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T11:57:04.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-30T11:57:04.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-30T11:57:04.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-30T11:57:04.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:57:04.213 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:57:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:57:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:57:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:57:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T11:57:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:57:04.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:57:04.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:57:04.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:57:04.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:57:04.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:57:04.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-30T11:57:04.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-30T11:57:04.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-30T11:57:04.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-30T11:57:04.261 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:57:04.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:04.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:57:04.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:04.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:57:04.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:04.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-30T11:57:04.809 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-30T11:54:35.305443+0000 last_deep_scrub_stamp 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:04.821 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:57:05.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:54:35.305443+0000 '>' 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:05.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:57:06.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:57:06.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:57:06.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:57:06.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:06.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:57:06.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:06.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:57:06.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:54:35.305443+0000 '>' 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:06.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:57:07.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:57:07.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:57:07.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:57:07.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:57:07.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:57:07.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:57:07.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:57:07.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:57:05.840841+0000 '>' 2024-03-30T11:54:35.305443+0000 2024-03-30T11:57:07.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:57:07.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-30T11:57:07.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-30T11:57:07.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-30T11:57:07.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T11:57:07.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-30T11:57:07.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-30T11:57:07.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-30T11:57:07.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-03-30T11:57:07.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-03-30T11:57:07.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:57:07.761 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:57:07.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:57:07.785 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-30T11:57:07.785 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T11:57:07.785 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T11:57:07.785 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T11:57:07.785 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T11:57:07.785 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-30T11:57:07.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T11:57:07.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:57:07.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T11:57:07.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-30T11:57:07.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-30T11:57:07.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-30T11:57:08.183 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' removed 2024-03-30T11:57:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T11:57:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:57:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:57:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:57:08.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:57:08.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:57:08.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:57:08.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:57:08.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:57:08.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:57:08.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:57:08.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:57:08.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:57:08.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:57:08.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:57:08.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:57:08.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:57:08.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:57:08.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:57:08.308 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:57:08.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:57:08.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:57:08.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:57:08.336 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:57:08.336 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:08.336 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:08.336 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:57:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:57:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:57:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T11:57:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T11:57:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T11:57:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T11:57:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:57:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:57:08.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T11:57:08.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T11:57:08.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T11:57:08.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T11:57:08.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T11:57:08.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T11:57:08.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:57:08.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T11:57:08.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T11:57:08.344 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T11:57:08.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T11:57:08.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T11:57:08.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T11:57:08.347 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T11:57:08.347 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:08.347 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:08.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T11:57:08.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T11:57:08.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T11:57:08.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T11:57:08.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T11:57:08.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:08.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:08.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T11:57:08.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T11:57:08.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T11:57:08.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T11:57:08.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T11:57:08.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T11:57:08.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-30T11:57:08.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T11:57:08.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:57:08.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:57:08.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:57:08.750 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:08.750 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:08.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:57:08.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-30T11:57:08.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:57:08.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:57:08.961 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:57:08.961 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:57:08.961 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:08.962 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:08.962 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:57:08.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:57:08.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T11:57:09.049 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:57:09.050 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:09.050 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:09.050 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T11:57:09.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T11:57:09.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T11:57:09.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-30T11:57:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T11:57:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T11:57:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T11:57:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T11:57:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T11:57:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T11:57:09.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T11:57:09.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:57:09.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:57:09.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:57:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:09.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:57:09.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:57:09.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T11:57:09.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:57:09.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:57:09.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:57:09.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:57:09.995 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:57:09.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0' 2024-03-30T11:57:09.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:57:10.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAW/gdmPI0JARAAeRFF/cYXVuVWukSOqd5c5A== 2024-03-30T11:57:10.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAW/gdmPI0JARAAeRFF/cYXVuVWukSOqd5c5A=="}' 2024-03-30T11:57:10.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 -i td/osd-scrub-repair/0/new.json 2024-03-30T11:57:10.216 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:57:10.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T11:57:10.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAW/gdmPI0JARAAeRFF/cYXVuVWukSOqd5c5A== --osd-uuid d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:57:10.254 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.262+0000 7f7a3d72a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:10.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.274+0000 7f7a3d72a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:10.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.275+0000 7f7a3d72a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:10.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.275+0000 7f7a3d72a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:57:10.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.275+0000 7f7a3d72a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:57:10.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.275+0000 7f7a3d72a640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T11:57:10.358 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:10.366+0000 7f7a3d72a640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T11:57:16.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T11:57:16.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:57:16.015 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T11:57:16.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T11:57:16.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:57:16.384 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T11:57:16.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T11:57:16.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:57:16.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:57:16.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:57:16.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:57:16.410 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:16.417+0000 7f85e6bac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:16.412 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:16.419+0000 7f85e6bac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:16.413 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:16.420+0000 7f85e6bac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:16.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T11:57:16.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:57:16.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:57:16.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:57:16.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:57:16.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:16.669 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:57:16.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:57:16.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:16.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:16.942 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:16.950+0000 7f85e6bac640 -1 Falling back to public interface 2024-03-30T11:57:16.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:17.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:17.951 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:57:17.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:17.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:57:17.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:17.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:18.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:19.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:19.248 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:57:19.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:19.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:57:19.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:19.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:19.272 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:19.280+0000 7f85e6bac640 -1 osd.0 0 log_to_monitors true 2024-03-30T11:57:19.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:20.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:20.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:20.709 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:57:20.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:57:20.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:20.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:21.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:22.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:22.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:22.009 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:57:22.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:57:22.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:22.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:22.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:23.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:23.303 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:57:23.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:23.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:57:23.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:23.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:23.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:24.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:24.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:24.598 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:57:24.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:57:24.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:24.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:24.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:25.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:25.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:25.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:57:25.891 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:57:25.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:25.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:26.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:27.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:27.238 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:57:27.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:27.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:57:27.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:27.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:27.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:28.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:28.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:28.530 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:57:28.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:57:28.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:28.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:28.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:29.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:29.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:29.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:57:29.844 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:57:29.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:29.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:30.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:31.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:31.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:31.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:57:31.196 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:57:31.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:31.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:31.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:32.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:32.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:32.631 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:57:32.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:57:32.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:32.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:33.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:34.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:34.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:34.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:57:34.113 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:57:34.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:34.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:34.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:35.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:35.783 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:57:35.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:35.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:57:35.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:35.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:36.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:37.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:37.334 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:57:37.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:37.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:57:37.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:37.335 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:37.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:38.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:38.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:38.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:57:38.905 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:57:38.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:38.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:39.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:40.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:40.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:40.202 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:57:40.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:57:40.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:40.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:41.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:42.050 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:57:42.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:57:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:57:42.345 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1977982273,v1:127.0.0.1:6803/1977982273] [v2:127.0.0.1:6804/1977982273,v1:127.0.0.1:6805/1977982273] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:57:42.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:57:42.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:57:42.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:57:42.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:57:42.351 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:57:42.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0708f773-8546-4cd5-8339-50541a603197' 2024-03-30T11:57:42.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T11:57:42.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2/gdm7TNJFhAAvvx9beqpapld+vwau7MPrw== 2024-03-30T11:57:42.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2/gdm7TNJFhAAvvx9beqpapld+vwau7MPrw=="}' 2024-03-30T11:57:42.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0708f773-8546-4cd5-8339-50541a603197 -i td/osd-scrub-repair/1/new.json 2024-03-30T11:57:42.700 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:57:42.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T11:57:42.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2/gdm7TNJFhAAvvx9beqpapld+vwau7MPrw== --osd-uuid 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:57:42.740 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.748+0000 7f83e47e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:42.784 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.792+0000 7f83e47e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:42.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.793+0000 7f83e47e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:42.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.793+0000 7f83e47e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:57:42.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.793+0000 7f83e47e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:57:42.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.793+0000 7f83e47e1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T11:57:42.859 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:42.866+0000 7f83e47e1640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T11:57:48.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T11:57:48.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T11:57:48.581 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T11:57:48.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T11:57:48.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T11:57:48.947 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T11:57:48.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T11:57:48.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:57:48.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T11:57:48.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T11:57:48.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T11:57:48.974 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:48.981+0000 7f388260d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:48.975 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:48.983+0000 7f388260d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:48.976 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:48.984+0000 7f388260d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:57:49.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T11:57:49.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:57:49.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:57:49.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:57:49.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:57:49.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:49.245 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T11:57:49.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:57:49.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:49.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:49.490 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:49.498+0000 7f388260d640 -1 Falling back to public interface 2024-03-30T11:57:49.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:50.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:50.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:50.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:57:50.529 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T11:57:50.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:50.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:50.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:51.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:51.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:51.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:57:51.820 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T11:57:51.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:51.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:51.841 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:57:51.848+0000 7f388260d640 -1 osd.1 0 log_to_monitors true 2024-03-30T11:57:52.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:53.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:53.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:53.150 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T11:57:53.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:57:53.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:53.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:53.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:54.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:54.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:54.514 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T11:57:54.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T11:57:54.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:54.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:54.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:55.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:55.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:55.900 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T11:57:55.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T11:57:55.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:55.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:56.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:57.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:57.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:57.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T11:57:57.268 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T11:57:57.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:57.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:57.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:58.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:58.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:58.681 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T11:57:58.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T11:57:58.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:58.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:57:58.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:57:59.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:57:59.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:57:59.980 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T11:57:59.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T11:57:59.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:57:59.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:00.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:01.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:01.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:01.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T11:58:01.292 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T11:58:01.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:01.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:01.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:02.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:02.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:02.699 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T11:58:02.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T11:58:02.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:02.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:02.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:03.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:03.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:03.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T11:58:03.992 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T11:58:03.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:03.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:04.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:05.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:05.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:05.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T11:58:05.407 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T11:58:05.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:05.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:05.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:06.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:06.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:06.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T11:58:06.880 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T11:58:06.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:06.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:07.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:08.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:08.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:08.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T11:58:08.506 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T11:58:08.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:08.506 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:08.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:09.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:09.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:09.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T11:58:09.822 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T11:58:09.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:09.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:10.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:11.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:11.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:11.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T11:58:11.936 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T11:58:11.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:11.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:12.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:13.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:13.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:13.478 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T11:58:13.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T11:58:13.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:13.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:14.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:58:15.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:58:15.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:58:15.140 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T11:58:15.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T11:58:15.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:58:15.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:58:15.430 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1430580451,v1:127.0.0.1:6811/1430580451] [v2:127.0.0.1:6812/1430580451,v1:127.0.0.1:6813/1430580451] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:58:15.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:58:15.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:58:15.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:58:15.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-03-30T11:58:15.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T11:58:15.703 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T11:58:15.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T11:58:15.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T11:58:16.292 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T11:58:16.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:58:17.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T11:58:19.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-30T11:58:19.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:58:19.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:58:19.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:58:19.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:58:19.053 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:58:19.053 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:58:19.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:58:19.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:58:19.053 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:58:19.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:58:19.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:58:19.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:58:19.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:58:19.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:58:19.151 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:58:19.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:58:19.435 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:58:19.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:58:19.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:58:19.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:58:19.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-03-30T11:58:19.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-03-30T11:58:19.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-03-30T11:58:19.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:58:19.540 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:58:19.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-03-30T11:58:19.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-03-30T11:58:19.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672964' 2024-03-30T11:58:19.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:58:19.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-03-30T11:58:19.646 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:58:19.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:58:19.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-03-30T11:58:19.647 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:58:19.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-03-30T11:58:19.648 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836491 2024-03-30T11:58:19.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-03-30T11:58:19.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:58:19.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-03-30T11:58:19.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:58:20.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:58:20.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:58:21.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-03-30T11:58:21.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:58:21.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-03-30T11:58:21.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:58:21.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:58:21.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-03-30T11:58:21.228 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:58:21.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-03-30T11:58:21.229 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672964 2024-03-30T11:58:21.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-03-30T11:58:21.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:58:21.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-03-30T11:58:21.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:58:21.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:58:21.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:58:21.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:58:22.147 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T11:58:22.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:58:22.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:58:22.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:58:22.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T11:58:22.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:58:22.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:58:22.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-03-30T11:58:22.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-30T11:58:22.887 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'foo' created 2024-03-30T11:58:22.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:58:23.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-03-30T11:58:23.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-30T11:58:24.409 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'csr_pool' created 2024-03-30T11:58:24.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T11:58:25.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-30T11:58:25.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:58:25.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:58:25.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:58:25.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:58:25.423 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:58:25.424 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:58:25.424 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:58:25.424 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:58:25.424 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:58:25.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:58:25.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:58:25.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:58:25.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:58:25.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:58:25.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:58:25.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:58:25.828 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:58:25.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:58:25.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:58:25.828 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:58:25.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-30T11:58:25.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-30T11:58:25.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-30T11:58:25.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:58:25.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:58:26.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-30T11:58:26.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-30T11:58:26.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672967' 2024-03-30T11:58:26.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:58:26.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-30T11:58:26.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:58:26.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:58:26.040 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-30T11:58:26.040 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:58:26.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-30T11:58:26.042 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836493 2024-03-30T11:58:26.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-30T11:58:26.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:58:26.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-30T11:58:26.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:58:26.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-30T11:58:26.323 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:58:26.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:58:26.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-30T11:58:26.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:58:26.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-30T11:58:26.325 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672967 2024-03-30T11:58:26.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-30T11:58:26.326 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:58:26.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-30T11:58:26.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:58:26.633 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:58:26.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:58:26.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:58:26.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:58:26.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:58:26.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:58:26.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:58:26.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:58:26.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:58:26.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:58:27.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:58:27.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:58:27.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:58:27.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:58:27.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:58:27.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:58:27.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:58:27.617 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:27.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:28.003 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:28.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:28.547 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:28.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:28.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:28.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:28.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-30T11:58:28.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:28.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:28.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:29.375 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:29.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:29.895 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:29.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:29.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:29.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:30.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-30T11:58:30.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:30.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:30.723 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:30.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:31.157 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:31.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:31.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:31.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:31.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-03-30T11:58:31.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:31.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:32.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:32.605 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:32.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:32.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:32.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:32.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-03-30T11:58:32.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-03-30T11:58:33.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:33.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:33.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:33.450 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:33.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:33.978 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:33.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:33.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:33.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:34.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-03-30T11:58:34.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-03-30T11:58:34.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:34.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-30T11:58:34.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-03-30T11:58:34.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:34.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-03-30T11:58:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:34.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:34.905 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:34.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:35.474 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:35.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:35.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:35.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:35.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-03-30T11:58:35.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-03-30T11:58:35.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:35.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-30T11:58:35.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-03-30T11:58:35.946 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:35.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:35.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-03-30T11:58:35.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:35.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:35.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:36.327 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:36.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:36.846 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:36.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:36.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:36.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:37.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-03-30T11:58:37.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:37.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:37.707 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:37.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:38.260 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:38.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:38.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:38.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:38.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-03-30T11:58:38.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-03-30T11:58:38.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:38.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-30T11:58:38.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-03-30T11:58:38.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:38.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:38.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-03-30T11:58:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:38.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:39.205 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:39.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:39.716 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:39.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:39.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:39.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:39.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-03-30T11:58:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-03-30T11:58:40.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:40.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-30T11:58:40.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-03-30T11:58:40.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:40.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-03-30T11:58:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:40.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:40.585 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:40.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:41.104 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:41.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:41.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:41.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:41.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-03-30T11:58:41.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-03-30T11:58:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-30T11:58:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-03-30T11:58:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:41.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-03-30T11:58:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:42.149 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:42.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:42.685 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:42.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:42.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:42.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:42.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-03-30T11:58:43.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:43.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:43.631 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:43.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:44.141 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:44.153 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:44.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:44.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:44.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-03-30T11:58:44.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:45.026 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:45.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:45.536 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:45.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:45.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:45.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:45.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-03-30T11:58:45.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-03-30T11:58:45.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:45.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-30T11:58:45.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-03-30T11:58:45.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:46.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:46.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-03-30T11:58:46.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:46.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:46.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:46.573 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:46.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:46.974 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:46.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:46.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:46.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:47.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-03-30T11:58:47.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-03-30T11:58:47.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:47.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:47.936 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:47.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:48.363 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:48.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:48.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:48.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:48.600 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-03-30T11:58:48.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-03-30T11:58:48.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:48.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:48.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:49.351 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:49.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:49.743 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:49.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:49.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:49.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:49.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-03-30T11:58:50.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-03-30T11:58:50.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:50.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:50.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:50.664 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:50.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:51.191 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:51.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:51.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:51.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:51.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-03-30T11:58:51.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-03-30T11:58:51.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:51.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:52.035 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:52.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:52.554 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:52.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:52.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:52.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:52.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-03-30T11:58:52.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T11:58:53.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T11:58:53.399 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T11:58:53.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T11:58:53.927 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T11:58:53.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T11:58:53.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T11:58:53.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-03-30T11:58:54.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-03-30T11:58:54.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-03-30T11:58:54.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-03-30T11:58:54.451 INFO:tasks.workunit.client.0.smithi003.stderr:1025+0 records in 2024-03-30T11:58:54.451 INFO:tasks.workunit.client.0.smithi003.stderr:1025+0 records out 2024-03-30T11:58:54.451 INFO:tasks.workunit.client.0.smithi003.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00178476 s, 588 MB/s 2024-03-30T11:58:54.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-03-30T11:58:54.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-03-30T11:58:54.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-30T11:58:54.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-30T11:58:54.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-30T11:58:54.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-30T11:58:54.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T11:58:54.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-30T11:58:54.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-03-30T11:58:54.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-30T11:58:54.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-30T11:58:54.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-30T11:58:54.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T11:58:55.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-03-30T11:58:55.212 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-03-30T11:58:55.212 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T11:58:55.212 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T11:58:55.213 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:58:55.213 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:58:55.213 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:58:55.213 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T11:58:55.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-30T11:58:55.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69973/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-30T11:58:55.292 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:58:55.292 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-30T11:58:55.293 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:58:55.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-03-30T11:58:55.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T11:58:55.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T11:58:55.300 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T11:58:55.300 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:58:55.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:58:55.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T11:58:55.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-30T11:58:55.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-30T11:58:55.381 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T11:58:55.381 INFO:tasks.workunit.client.0.smithi003.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-30T11:58:55.381 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T11:58:55.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-30T11:58:55.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-30T11:58:55.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:58:55.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:58:55.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:58:55.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:58:55.389 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:58:55.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-30T11:58:24.305245+0000 2024-03-30T11:58:55.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-30T11:58:55.932 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-30T11:58:55.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-30T11:58:24.305245+0000 last_deep_scrub_stamp 2024-03-30T11:58:55.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:58:24.305245+0000 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:58:55.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:58:56.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:24.305245+0000 '>' 2024-03-30T11:58:24.305245+0000 2024-03-30T11:58:56.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:58:57.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:58:57.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:58:57.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:58:57.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:58:57.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:58:57.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:58:57.223 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:58:57.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:24.305245+0000 '>' 2024-03-30T11:58:24.305245+0000 2024-03-30T11:58:57.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:58:58.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:58:58.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:58:58.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:58:58.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:58:58.512 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:58:58.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:58:58.513 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:58:58.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:24.305245+0000 '>' 2024-03-30T11:58:24.305245+0000 2024-03-30T11:58:58.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:58:59.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:58:59.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:58:59.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:58:59.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:58:59.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:58:59.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:58:59.805 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:59:00.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:24.305245+0000 '>' 2024-03-30T11:58:24.305245+0000 2024-03-30T11:59:00.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:59:01.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:59:01.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:59:01.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:59:01.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:59:01.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:59:01.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:59:01.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:59:01.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:24.305245+0000 '>' 2024-03-30T11:58:24.305245+0000 2024-03-30T11:59:01.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T11:59:02.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T11:59:02.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T11:59:02.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T11:59:02.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T11:59:02.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T11:59:02.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T11:59:02.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T11:59:02.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:57.119155+0000 '>' 2024-03-30T11:58:24.305245+0000 2024-03-30T11:59:02.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T11:59:02.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-30T11:59:02.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:59:02.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-30T11:59:02.678 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:02.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:59:02.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:59:02.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:04.206 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-03-30T11:59:04.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:59:04.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:59:04.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:59:04.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:59:04.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:59:04.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:59:04.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:59:04.828 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:04.835+0000 7ff468d6d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:04.830 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:04.837+0000 7ff468d6d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:04.831 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:04.839+0000 7ff468d6d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:05.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:05.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:05.349 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:05.357+0000 7ff468d6d640 -1 Falling back to public interface 2024-03-30T11:59:05.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:06.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:06.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:06.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:59:06.378 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:59:06.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:06.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:06.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:59:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:59:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:07.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:07.738 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:07.744+0000 7ff468d6d640 -1 osd.1 63 log_to_monitors true 2024-03-30T11:59:07.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:08.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:08.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:08.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:59:08.969 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:59:08.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:08.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [10,63) [v2:127.0.0.1:6810/4017425895,v1:127.0.0.1:6811/4017425895] [v2:127.0.0.1:6812/4017425895,v1:127.0.0.1:6813/4017425895] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:59:09.314 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:59:09.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:59:09.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:59:09.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:59:09.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:59:09.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:59:09.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:59:09.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:59:09.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:59:09.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:59:09.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:59:09.688 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:59:09.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:59:09.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:09.688 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:59:09.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-30T11:59:09.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-30T11:59:09.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-30T11:59:09.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:09.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:59:09.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-03-30T11:59:09.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-03-30T11:59:09.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-287762808834' 2024-03-30T11:59:09.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:09.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-30T11:59:09.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:09.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:59:09.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-30T11:59:09.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:09.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-30T11:59:09.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-30T11:59:09.902 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836503 2024-03-30T11:59:09.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:10.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-30T11:59:10.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:59:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:59:11.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:11.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-30T11:59:11.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:11.479 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808834 2024-03-30T11:59:11.479 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:11.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:59:11.480 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808834 2024-03-30T11:59:11.481 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:11.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-03-30T11:59:11.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2024-03-30T11:59:11.482 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 287762808834 2024-03-30T11:59:11.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:59:11.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-03-30T11:59:11.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:59:11.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:11.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:12.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:59:12.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:59:12.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:59:12.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:59:12.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:59:12.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:59:12.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:59:12.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:59:12.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:59:12.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:59:12.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:12.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:59:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:59:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:59:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:59:12.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-30T11:59:12.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-03-30T11:59:12.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T11:59:12.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:59:12.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:59:12.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:59:12.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:59:13.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:14.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:59:14.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:59:14.992 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:59:14.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:59:14.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:59:14.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:59:14.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:59:15.022 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:15.029+0000 7f8b68119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:15.024 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:15.031+0000 7f8b68119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:15.025 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:15.033+0000 7f8b68119640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:15.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:59:15.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:59:15.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:59:15.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:59:15.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:59:15.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:15.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:59:15.288 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:59:15.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:15.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:15.544 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:15.552+0000 7f8b68119640 -1 Falling back to public interface 2024-03-30T11:59:15.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:16.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:16.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:16.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:59:16.571 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:59:16.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:16.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:16.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:17.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:59:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:59:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:17.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:17.872 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:17.880+0000 7f8b68119640 -1 osd.0 68 log_to_monitors true 2024-03-30T11:59:18.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:19.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:19.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:19.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:59:19.193 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:59:19.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:19.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/2041920056,v1:127.0.0.1:6803/2041920056] [v2:127.0.0.1:6804/2041920056,v1:127.0.0.1:6805/2041920056] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:59:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:59:19.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:59:19.487 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:59:19.487 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:59:19.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:59:19.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:59:19.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:59:19.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:59:19.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:59:19.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:59:19.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:59:19.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:59:19.574 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:59:19.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:59:19.855 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:59:19.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:59:19.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:19.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:59:19.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-30T11:59:19.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-30T11:59:19.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314' 2024-03-30T11:59:19.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:19.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:59:20.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-03-30T11:59:20.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-03-30T11:59:20.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314 1-287762808837' 2024-03-30T11:59:20.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:20.066 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645314 2024-03-30T11:59:20.066 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:20.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:59:20.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645314 2024-03-30T11:59:20.067 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:20.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-30T11:59:20.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2024-03-30T11:59:20.069 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 309237645314 2024-03-30T11:59:20.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:20.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-03-30T11:59:20.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:20.367 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-287762808837 2024-03-30T11:59:20.367 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:20.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:59:20.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-287762808837 2024-03-30T11:59:20.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:20.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-03-30T11:59:20.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 287762808837' 2024-03-30T11:59:20.369 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 287762808837 2024-03-30T11:59:20.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:59:20.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-03-30T11:59:20.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:59:20.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:20.651 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:21.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:59:21.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:59:21.003 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:59:21.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:59:21.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:59:21.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:59:21.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:59:21.004 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:59:21.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:59:21.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:59:21.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:21.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:21.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:59:21.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:59:21.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:59:21.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:59:21.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-30T11:59:21.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-03-30T11:59:21.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T11:59:21.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:59:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:59:21.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:59:21.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:59:21.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:59:21.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-03-30T11:59:21.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:59:21.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:59:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:59:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:59:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:59:21.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-03-30T11:59:23.230 INFO:tasks.workunit.client.0.smithi003.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-03-30T11:59:23.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:59:23.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:59:23.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:59:23.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:59:23.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:59:23.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:59:23.777 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:23.784+0000 7fbf56193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:23.779 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:23.787+0000 7fbf56193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:23.780 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:23.788+0000 7fbf56193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:24.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:24.294 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:24.302+0000 7fbf56193640 -1 Falling back to public interface 2024-03-30T11:59:24.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:25.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:25.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:25.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:59:25.344 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:59:25.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:25.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:25.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:26.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:26.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:26.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:59:26.637 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:59:26.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:26.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:26.713 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:26.721+0000 7fbf56193640 -1 osd.1 73 log_to_monitors true 2024-03-30T11:59:26.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:27.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:27.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:27.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:59:27.932 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:59:27.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:27.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:28.229 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [67,73) [v2:127.0.0.1:6810/600318147,v1:127.0.0.1:6811/600318147] [v2:127.0.0.1:6812/600318147,v1:127.0.0.1:6813/600318147] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:59:28.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:59:28.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:59:28.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:59:28.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:59:28.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:59:28.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:59:28.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:59:28.316 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:59:28.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:59:28.596 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:59:28.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:59:28.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:28.596 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:59:28.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645317 2024-03-30T11:59:28.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645317 2024-03-30T11:59:28.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317' 2024-03-30T11:59:28.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:28.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:59:28.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-30T11:59:28.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-30T11:59:28.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317 1-330712481794' 2024-03-30T11:59:28.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:28.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645317 2024-03-30T11:59:28.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:28.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:59:28.809 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645317 2024-03-30T11:59:28.809 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:28.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645317 2024-03-30T11:59:28.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645317' 2024-03-30T11:59:28.810 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 309237645317 2024-03-30T11:59:28.810 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:29.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645316 -lt 309237645317 2024-03-30T11:59:29.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:59:30.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:59:30.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:30.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645317 -lt 309237645317 2024-03-30T11:59:30.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:30.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-30T11:59:30.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:30.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:59:30.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-30T11:59:30.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:30.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-30T11:59:30.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-30T11:59:30.411 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 330712481794 2024-03-30T11:59:30.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:59:30.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-30T11:59:30.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:59:30.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:30.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:31.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:59:31.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:59:31.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:59:31.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:59:31.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:59:31.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:59:31.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:59:31.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:59:31.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:59:31.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:59:31.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:31.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:31.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:59:31.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:59:31.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:59:31.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:59:31.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-30T11:59:31.717 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:59:31.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:59:31.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:59:31.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:59:31.720 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:59:31.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:59:31.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:59:31.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:59:32.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-30T11:59:33.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:59:33.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:59:33.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:59:33.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:59:33.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:59:33.816 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:59:33.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:59:33.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:59:33.818 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:59:33.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:59:33.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:59:33.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:59:33.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:59:33.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:59:33.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:59:33.846 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:33.853+0000 7f563aeb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:33.847 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:33.855+0000 7f563aeb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:33.848 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:33.856+0000 7f563aeb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:34.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:34.370 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:34.378+0000 7f563aeb5640 -1 Falling back to public interface 2024-03-30T11:59:34.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:35.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:35.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:35.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:59:35.388 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:59:35.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:35.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:35.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:36.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:36.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:36.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:59:36.683 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:59:36.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:36.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:36.734 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:36.742+0000 7f563aeb5640 -1 osd.0 78 log_to_monitors true 2024-03-30T11:59:37.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:38.021 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:38.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:38.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:59:38.022 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:59:38.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:38.022 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:38.382 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/1497569113,v1:127.0.0.1:6803/1497569113] [v2:127.0.0.1:6804/1497569113,v1:127.0.0.1:6805/1497569113] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:59:38.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:59:38.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:59:38.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:59:38.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:59:38.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:59:38.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:59:38.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:59:38.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:59:38.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:59:38.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:59:38.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:59:38.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:59:38.750 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:59:38.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:59:38.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:38.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:59:38.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-30T11:59:38.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-30T11:59:38.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-03-30T11:59:38.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:38.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:59:38.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-03-30T11:59:38.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-03-30T11:59:38.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-03-30T11:59:38.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:38.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-03-30T11:59:38.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:38.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:59:38.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-03-30T11:59:38.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:38.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-30T11:59:38.963 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-03-30T11:59:38.963 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 352187318274 2024-03-30T11:59:38.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:39.245 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-03-30T11:59:39.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:59:40.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:59:40.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:40.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-30T11:59:40.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:40.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-03-30T11:59:40.569 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:40.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:59:40.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-03-30T11:59:40.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:40.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-03-30T11:59:40.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-03-30T11:59:40.572 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 330712481797 2024-03-30T11:59:40.572 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:59:40.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-03-30T11:59:40.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:59:40.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:40.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:59:41.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:59:41.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:59:41.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:59:41.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:59:41.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:59:41.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:59:41.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:59:41.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:59:41.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:59:41.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:41.498 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:59:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:59:41.854 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:59:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:59:41.855 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-30T11:59:41.855 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-03-30T11:59:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T11:59:41.856 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:59:41.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:59:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:59:41.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:59:42.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-03-30T11:59:44.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:59:44.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:59:44.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:59:44.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:59:44.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:59:44.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:59:44.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:59:44.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:59:44.076 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:44.083+0000 7f24f5fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:44.078 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:44.086+0000 7f24f5fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:44.080 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:44.087+0000 7f24f5fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:44.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:59:44.343 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:59:44.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:44.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:44.594 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:44.602+0000 7f24f5fac640 -1 Falling back to public interface 2024-03-30T11:59:44.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:45.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:45.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:45.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:59:45.647 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:59:45.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:45.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:45.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:46.935 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:46.942+0000 7f24f5fac640 -1 osd.1 84 log_to_monitors true 2024-03-30T11:59:46.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:46.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:46.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:59:46.943 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:59:46.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:46.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:47.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:59:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:59:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:48.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T11:59:48.674 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/3259163307,v1:127.0.0.1:6811/3259163307] [v2:127.0.0.1:6812/3259163307,v1:127.0.0.1:6813/3259163307] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T11:59:48.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:59:48.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:59:48.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:59:48.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:59:48.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:59:48.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:59:48.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:59:48.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:59:48.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:59:48.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:59:48.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:59:48.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:59:48.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:59:49.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:59:49.043 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:59:49.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:59:49.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:49.043 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T11:59:49.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-03-30T11:59:49.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-03-30T11:59:49.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-03-30T11:59:49.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:49.148 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T11:59:49.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-30T11:59:49.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-30T11:59:49.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-03-30T11:59:49.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:49.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-03-30T11:59:49.255 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:49.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T11:59:49.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-03-30T11:59:49.256 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:49.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-03-30T11:59:49.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-03-30T11:59:49.257 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 352187318277 2024-03-30T11:59:49.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:49.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-03-30T11:59:49.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T11:59:50.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T11:59:50.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T11:59:50.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318277 -lt 352187318277 2024-03-30T11:59:50.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T11:59:50.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-03-30T11:59:50.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T11:59:50.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T11:59:50.837 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-03-30T11:59:50.837 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T11:59:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-30T11:59:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-03-30T11:59:50.838 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 377957122050 2024-03-30T11:59:50.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T11:59:51.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-30T11:59:51.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T11:59:51.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:51.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T11:59:51.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T11:59:51.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T11:59:51.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T11:59:51.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T11:59:51.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T11:59:52.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T11:59:52.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T11:59:52.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T11:59:52.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T11:59:52.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-30T11:59:52.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-03-30T11:59:52.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T11:59:52.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T11:59:52.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-03-30T11:59:52.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T11:59:52.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:59:53.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-30T11:59:54.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T11:59:54.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T11:59:54.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T11:59:54.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T11:59:54.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T11:59:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T11:59:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T11:59:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T11:59:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T11:59:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T11:59:54.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T11:59:54.972 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T11:59:54.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T11:59:54.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T11:59:54.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T11:59:54.972 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T11:59:54.998 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:55.005+0000 7f43c440f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:55.000 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:55.007+0000 7f43c440f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:55.001 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:55.009+0000 7f43c440f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T11:59:55.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T11:59:55.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T11:59:55.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:55.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:55.516 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:55.523+0000 7f43c440f640 -1 Falling back to public interface 2024-03-30T11:59:55.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:56.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:56.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:56.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T11:59:56.546 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T11:59:56.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:56.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:56.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:57.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:57.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:57.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T11:59:57.842 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T11:59:57.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:57.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:57.888 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T11:59:57.894+0000 7f43c440f640 -1 osd.0 89 log_to_monitors true 2024-03-30T11:59:58.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T11:59:59.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T11:59:59.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T11:59:59.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T11:59:59.167 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T11:59:59.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T11:59:59.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T11:59:59.532 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/1138390894,v1:127.0.0.1:6803/1138390894] [v2:127.0.0.1:6804/1138390894,v1:127.0.0.1:6805/1138390894] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T11:59:59.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T11:59:59.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T11:59:59.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T11:59:59.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T11:59:59.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T11:59:59.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T11:59:59.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T11:59:59.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T11:59:59.534 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T11:59:59.534 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T11:59:59.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T11:59:59.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T11:59:59.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T11:59:59.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T11:59:59.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T11:59:59.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T11:59:59.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T11:59:59.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T11:59:59.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T11:59:59.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T11:59:59.903 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T11:59:59.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T11:59:59.903 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T11:59:59.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:00:00.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-30T12:00:00.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-30T12:00:00.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-03-30T12:00:00.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:00.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:00:00.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-03-30T12:00:00.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-03-30T12:00:00.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-03-30T12:00:00.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:00.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-03-30T12:00:00.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:00.114 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:00:00.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-03-30T12:00:00.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:00.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-30T12:00:00.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-03-30T12:00:00.116 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 399431958530 2024-03-30T12:00:00.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:00.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-03-30T12:00:00.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:00.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-03-30T12:00:00.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:00.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:00:00.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-03-30T12:00:00.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:00.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-03-30T12:00:00.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-03-30T12:00:00.430 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 377957122053 2024-03-30T12:00:00.430 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:00:00.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122053 -lt 377957122053 2024-03-30T12:00:00.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:00:00.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:00.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:01.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:00:01.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:01.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:01.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:01.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:01.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:01.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:01.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:01.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:00:01.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:01.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:01.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:01.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:00:01.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:00:01.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:00:01.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-03-30T12:00:01.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:00:01.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-30T12:00:01.735 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-03-30T12:00:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:00:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:00:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-03-30T12:00:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-30T12:00:01.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:00:01.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:02.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-30T12:00:03.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:00:03.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:00:03.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:00:03.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:00:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:00:03.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:00:03.923 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:03.931+0000 7efc08015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:03.925 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:03.933+0000 7efc08015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:03.927 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:03.934+0000 7efc08015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:04.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:04.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:04.445 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:04.453+0000 7efc08015640 -1 Falling back to public interface 2024-03-30T12:00:04.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:05.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:00:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:00:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:05.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:06.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:06.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:06.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:00:06.777 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:00:06.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:06.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:06.814 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:06.822+0000 7efc08015640 -1 osd.1 95 log_to_monitors true 2024-03-30T12:00:07.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:08.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:08.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:08.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:00:08.176 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:00:08.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:08.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:08.550 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/2738581418,v1:127.0.0.1:6811/2738581418] [v2:127.0.0.1:6812/2738581418,v1:127.0.0.1:6813/2738581418] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:00:08.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:00:08.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:00:08.551 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:00:08.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:00:08.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:00:08.552 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:00:08.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:00:08.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:00:08.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:00:08.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:00:08.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:00:08.638 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:00:08.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:00:08.918 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:00:08.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:00:08.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:08.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:00:09.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-03-30T12:00:09.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-03-30T12:00:09.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-03-30T12:00:09.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:09.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:00:09.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-30T12:00:09.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-30T12:00:09.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-03-30T12:00:09.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:09.129 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-03-30T12:00:09.129 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:09.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:00:09.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-03-30T12:00:09.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-03-30T12:00:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-03-30T12:00:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 399431958533 2024-03-30T12:00:09.132 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:09.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958531 -lt 399431958533 2024-03-30T12:00:09.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:00:10.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:00:10.414 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:10.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958533 -lt 399431958533 2024-03-30T12:00:10.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:10.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-03-30T12:00:10.709 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:10.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:00:10.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-03-30T12:00:10.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:10.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-30T12:00:10.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-03-30T12:00:10.712 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 425201762306 2024-03-30T12:00:10.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:00:10.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-30T12:00:10.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:00:10.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:10.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:11.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:00:11.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:11.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:11.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:11.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:11.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:11.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:11.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:11.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:00:11.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:11.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:11.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:11.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:00:11.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:00:11.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:00:11.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-03-30T12:00:11.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:00:11.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-30T12:00:11.977 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-03-30T12:00:11.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T12:00:11.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:00:11.978 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-03-30T12:00:12.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:00:12.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:00:12.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:00:12.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:00:12.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-30T12:00:14.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:00:14.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:00:14.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:00:14.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:00:14.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:00:14.202 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:00:14.203 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:00:14.203 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:00:14.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:14.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:00:14.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:00:14.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:00:14.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:14.240+0000 7eff6ccd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:14.235 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:14.243+0000 7eff6ccd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:14.236 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:14.244+0000 7eff6ccd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:14.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:00:14.755 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:14.762+0000 7eff6ccd9640 -1 Falling back to public interface 2024-03-30T12:00:14.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:00:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:00:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:15.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:00:16.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:17.046 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:17.053+0000 7eff6ccd9640 -1 osd.0 101 log_to_monitors true 2024-03-30T12:00:17.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:17.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:17.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:00:17.079 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:00:17.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:17.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:00:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 105 up_thru 100 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/951714179,v1:127.0.0.1:6803/951714179] [v2:127.0.0.1:6804/951714179,v1:127.0.0.1:6805/951714179] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:00:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:00:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:00:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:00:17.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:00:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:00:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:00:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:00:17.461 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:00:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:00:17.461 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:00:17.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:00:17.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:00:17.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:00:17.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:00:17.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:00:17.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:00:17.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:00:17.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:00:17.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:00:17.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:00:17.857 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:00:17.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:00:17.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:17.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:00:17.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-30T12:00:17.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-30T12:00:17.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-03-30T12:00:17.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:17.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:00:18.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-03-30T12:00:18.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-03-30T12:00:18.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-03-30T12:00:18.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:18.096 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-03-30T12:00:18.096 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:18.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:00:18.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-03-30T12:00:18.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:18.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-30T12:00:18.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-03-30T12:00:18.099 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 450971566082 2024-03-30T12:00:18.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:18.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-03-30T12:00:18.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:18.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762309 2024-03-30T12:00:18.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:18.410 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:00:18.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762309 2024-03-30T12:00:18.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:18.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-03-30T12:00:18.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-03-30T12:00:18.412 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 425201762309 2024-03-30T12:00:18.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:00:18.692 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762309 -lt 425201762309 2024-03-30T12:00:18.692 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:00:18.692 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:18.693 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:19.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:19.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:00:19.365 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:19.365 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:19.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:00:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T12:00:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T12:00:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T12:00:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:00:19.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T12:00:19.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:00:19.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:19.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:20.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:00:20.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:20.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:20.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:00:20.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:00:20.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:00:20.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:00:20.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:00:20.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:00:20.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T12:00:20.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:00:20.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T12:00:21.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:00:21.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:21.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:21.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:21.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:21.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:21.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:21.015 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:21.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:00:21.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:21.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:21.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:21.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:00:21.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:00:22.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:00:22.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:00:22.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T12:00:22.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:00:22.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T12:00:22.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:00:22.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:22.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:22.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:22.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:22.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:22.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:22.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:22.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:00:22.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:22.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:22.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:00:23.040 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:00:23.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:00:23.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:00:23.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-30T12:00:23.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:00:23.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-30T12:00:24.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:00:24.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:24.194 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:24.194 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:24.194 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:24.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:24.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:24.195 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:24.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:00:24.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:24.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:24.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:00:24.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:00:24.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:25.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-30T12:00:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:00:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:00:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:00:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:00:26.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:00:26.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:00:26.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:00:26.881 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:00:26.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:00:26.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:00:26.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:00:26.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:00:26.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:00:26.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:00:26.907 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:26.915+0000 7fe72268b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:26.910 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:26.918+0000 7fe72268b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:26.911 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:26.919+0000 7fe72268b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:27.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:27.427 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:27.435+0000 7fe72268b640 -1 Falling back to public interface 2024-03-30T12:00:27.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:28.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:28.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:28.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:00:28.455 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:00:28.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:28.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:28.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:29.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:29.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:29.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:00:29.752 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:00:29.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:29.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:29.826 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:29.834+0000 7fe72268b640 -1 osd.1 106 log_to_monitors true 2024-03-30T12:00:30.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:31.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:31.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:31.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:00:31.097 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:00:31.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:31.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:31.411 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/1658476529,v1:127.0.0.1:6811/1658476529] [v2:127.0.0.1:6812/1658476529,v1:127.0.0.1:6813/1658476529] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:00:31.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:00:31.412 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:00:31.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:00:31.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:00:31.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:00:31.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:00:31.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:00:31.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:00:31.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:00:31.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:00:31.499 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:00:31.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:00:31.778 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:00:31.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:00:31.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:31.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:00:31.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-03-30T12:00:31.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-03-30T12:00:31.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-03-30T12:00:31.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:31.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:00:31.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=472446402562 2024-03-30T12:00:31.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 472446402562 2024-03-30T12:00:31.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-472446402562' 2024-03-30T12:00:31.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:31.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-03-30T12:00:31.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:31.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:00:31.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-03-30T12:00:31.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:31.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-03-30T12:00:31.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-03-30T12:00:31.997 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 450971566085 2024-03-30T12:00:31.998 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:32.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-03-30T12:00:32.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:32.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-472446402562 2024-03-30T12:00:32.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:32.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:00:32.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-472446402562 2024-03-30T12:00:32.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:32.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=472446402562 2024-03-30T12:00:32.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2024-03-30T12:00:32.283 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 472446402562 2024-03-30T12:00:32.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:00:32.576 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 472446402562 -lt 472446402562 2024-03-30T12:00:32.577 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:00:32.577 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:32.577 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:32.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:32.931 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:33.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:00:33.207 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:33.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:33.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:00:33.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:00:33.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:00:33.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:33.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-30T12:00:35.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:00:35.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:00:35.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:00:35.922 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:00:35.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:00:35.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:00:35.924 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:00:35.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:00:35.925 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:00:35.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:00:35.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:00:35.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:00:35.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:00:35.951 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:35.959+0000 7f675551f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:35.953 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:35.961+0000 7f675551f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:35.954 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:35.962+0000 7f675551f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:36.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:00:36.243 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:00:36.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:36.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:36.475 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:36.483+0000 7f675551f640 -1 Falling back to public interface 2024-03-30T12:00:36.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:37.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:37.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:37.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:00:37.523 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:00:37.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:37.523 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:37.816 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:38.796 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:38.804+0000 7f675551f640 -1 osd.1 112 log_to_monitors true 2024-03-30T12:00:38.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:38.817 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:38.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:00:38.818 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:00:38.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:38.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:39.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:00:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:00:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:40.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:40.431 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/3839450582,v1:127.0.0.1:6811/3839450582] [v2:127.0.0.1:6812/3839450582,v1:127.0.0.1:6813/3839450582] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:00:40.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:00:40.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:00:40.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:00:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:00:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:00:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:00:40.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:00:40.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:00:40.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:00:40.877 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:00:40.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:00:40.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:40.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:00:40.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-03-30T12:00:40.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-03-30T12:00:40.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088' 2024-03-30T12:00:40.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:40.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:00:41.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-30T12:00:41.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-30T12:00:41.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088 1-498216206338' 2024-03-30T12:00:41.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:41.088 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566088 2024-03-30T12:00:41.089 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:41.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:00:41.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566088 2024-03-30T12:00:41.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:41.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-03-30T12:00:41.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-03-30T12:00:41.091 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 450971566088 2024-03-30T12:00:41.091 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566087 -lt 450971566088 2024-03-30T12:00:41.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:00:42.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:00:42.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:42.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566088 2024-03-30T12:00:42.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:42.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-498216206338 2024-03-30T12:00:42.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:42.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:00:42.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-498216206338 2024-03-30T12:00:42.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:42.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-30T12:00:42.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2024-03-30T12:00:42.668 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 498216206338 2024-03-30T12:00:42.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:00:42.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-03-30T12:00:42.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:00:42.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:42.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:43.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:43.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:00:43.583 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:43.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:43.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:43.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:00:43.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:00:43.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:00:43.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-30T12:00:43.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:00:43.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-30T12:00:43.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:00:43.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:00:43.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:00:43.943 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:00:43.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:00:43.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:00:43.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:00:44.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:00:44.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-30T12:00:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:00:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:00:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:00:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:00:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:44.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:46.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:00:46.071 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:00:46.072 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:00:46.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:00:46.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:00:46.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:00:46.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:00:46.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:00:46.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:00:46.098 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:46.105+0000 7fb5e8ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:46.100 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:46.107+0000 7fb5e8ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:46.101 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:46.109+0000 7fb5e8ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:46.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:00:46.367 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:00:46.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:46.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:46.619 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:46.627+0000 7fb5e8ec6640 -1 Falling back to public interface 2024-03-30T12:00:46.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:47.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:47.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:47.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:00:47.661 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:00:47.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:47.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:47.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:48.901 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:48.909+0000 7fb5e8ec6640 -1 osd.1 117 log_to_monitors true 2024-03-30T12:00:48.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:48.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:48.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:00:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:00:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:48.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:49.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:50.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:00:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:00:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/1315177547,v1:127.0.0.1:6811/1315177547] [v2:127.0.0.1:6812/1315177547,v1:127.0.0.1:6813/1315177547] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:00:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:00:50.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:00:50.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:00:50.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:00:50.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:00:50.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:00:50.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:00:50.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:00:51.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:00:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:00:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:00:51.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:51.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:00:51.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566091 2024-03-30T12:00:51.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566091 2024-03-30T12:00:51.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091' 2024-03-30T12:00:51.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:00:51.155 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:00:51.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-03-30T12:00:51.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-03-30T12:00:51.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091 1-519691042818' 2024-03-30T12:00:51.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:51.261 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566091 2024-03-30T12:00:51.261 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:51.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:00:51.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566091 2024-03-30T12:00:51.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:51.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566091 2024-03-30T12:00:51.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-03-30T12:00:51.264 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 450971566091 2024-03-30T12:00:51.264 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566091 2024-03-30T12:00:51.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:00:52.550 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:00:52.550 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:00:52.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566091 -lt 450971566091 2024-03-30T12:00:52.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:00:52.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-519691042818 2024-03-30T12:00:52.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:00:52.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:00:52.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-519691042818 2024-03-30T12:00:52.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:00:52.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-03-30T12:00:52.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2024-03-30T12:00:52.844 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 519691042818 2024-03-30T12:00:52.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:00:53.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-03-30T12:00:53.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:00:53.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:53.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:00:53.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:00:53.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:00:53.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:00:53.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:00:53.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:00:54.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:00:54.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:00:54.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:00:54.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-03-30T12:00:54.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:00:54.283 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:00:54.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:00:54.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:00:54.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:00:54.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:54.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:00:56.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:00:56.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:00:56.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:00:56.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:00:56.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:00:56.442 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:00:56.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:00:56.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:00:56.444 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:00:56.444 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:00:56.445 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:00:56.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:00:56.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:00:56.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:00:56.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:00:56.471 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:56.478+0000 7f992d60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:56.473 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:56.481+0000 7f992d60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:56.475 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:56.482+0000 7f992d60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:00:56.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:00:56.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:00:56.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:00:56.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:00:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:00:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:00:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:00:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:57.004 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:57.012+0000 7f992d60f640 -1 Falling back to public interface 2024-03-30T12:00:57.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:00:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:00:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:58.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:58.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:00:59.312 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:00:59.320+0000 7f992d60f640 -1 osd.1 123 log_to_monitors true 2024-03-30T12:00:59.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:00:59.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:00:59.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:00:59.320 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:00:59.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:00:59.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:00:59.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:00.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:00.650 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:00.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:01:00.651 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:01:00.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:00.651 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:01.019 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/3484614556,v1:127.0.0.1:6811/3484614556] [v2:127.0.0.1:6812/3484614556,v1:127.0.0.1:6813/3484614556] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:01:01.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:01:01.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:01:01.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:01:01.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:01:01.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:01:01.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:01:01.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:01:01.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:01:01.391 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:01:01.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:01:01.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:01.393 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:01.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-03-30T12:01:01.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-03-30T12:01:01.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094' 2024-03-30T12:01:01.496 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:01.496 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:01.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846594 2024-03-30T12:01:01.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846594 2024-03-30T12:01:01.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094 1-545460846594' 2024-03-30T12:01:01.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:01.602 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566094 2024-03-30T12:01:01.602 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:01.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:01:01.603 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566094 2024-03-30T12:01:01.603 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:01.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-03-30T12:01:01.604 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-03-30T12:01:01.604 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 450971566094 2024-03-30T12:01:01.605 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:01.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566093 -lt 450971566094 2024-03-30T12:01:01.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:01:02.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:01:02.888 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:03.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566094 2024-03-30T12:01:03.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:03.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846594 2024-03-30T12:01:03.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:03.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:01:03.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846594 2024-03-30T12:01:03.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:03.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846594 2024-03-30T12:01:03.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2024-03-30T12:01:03.186 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 545460846594 2024-03-30T12:01:03.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:01:03.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846594 -lt 545460846594 2024-03-30T12:01:03.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:01:03.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:03.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:01:03.818 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:01:04.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:01:04.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:01:04.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:04.098 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:04.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:01:04.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:01:04.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:01:04.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-30T12:01:04.455 INFO:tasks.workunit.client.0.smithi003.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-30T12:01:04.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:04.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-30T12:01:04.455 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-03-30T12:01:04.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T12:01:04.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:04.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:04.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-30T12:01:04.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-03-30T12:01:04.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:01:04.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:04.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:04.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-30T12:01:04.457 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-03-30T12:01:04.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T12:01:04.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:04.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:04.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-30T12:01:04.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-03-30T12:01:04.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:01:04.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:04.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:04.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-30T12:01:04.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:01:04.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:01:04.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:01:04.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:01:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-30T12:01:06.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:01:06.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:01:06.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:01:06.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:01:06.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:01:06.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:01:06.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:01:06.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:01:06.469 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:01:06.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:01:06.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:01:06.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:01:06.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:01:06.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:01:06.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:01:06.496 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:06.503+0000 7f1006e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:06.498 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:06.506+0000 7f1006e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:06.499 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:06.507+0000 7f1006e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:06.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:07.013 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:07.021+0000 7f1006e1c640 -1 Falling back to public interface 2024-03-30T12:01:07.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:08.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:08.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:08.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:01:08.043 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:01:08.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:08.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:09.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:09.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:09.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:01:09.340 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:01:09.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:09.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:09.482 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:09.489+0000 7f1006e1c640 -1 osd.0 129 log_to_monitors true 2024-03-30T12:01:09.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:01:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:01:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:10.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/2969059860,v1:127.0.0.1:6803/2969059860] [v2:127.0.0.1:6804/2969059860,v1:127.0.0.1:6805/2969059860] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:01:11.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:01:11.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:01:11.078 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:01:11.078 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:01:11.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:01:11.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:01:11.078 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:01:11.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:01:11.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:01:11.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:01:11.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:01:11.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:01:11.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:01:11.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:01:11.468 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:01:11.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:01:11.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:11.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:11.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650370 2024-03-30T12:01:11.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650370 2024-03-30T12:01:11.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370' 2024-03-30T12:01:11.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:11.574 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:11.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=545460846597 2024-03-30T12:01:11.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 545460846597 2024-03-30T12:01:11.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650370 1-545460846597' 2024-03-30T12:01:11.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:11.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650370 2024-03-30T12:01:11.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:11.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:01:11.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650370 2024-03-30T12:01:11.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:11.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650370 2024-03-30T12:01:11.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2024-03-30T12:01:11.685 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 571230650370 2024-03-30T12:01:11.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:11.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 571230650370 2024-03-30T12:01:11.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:01:12.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:01:12.967 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:13.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650370 -lt 571230650370 2024-03-30T12:01:13.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:13.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-545460846597 2024-03-30T12:01:13.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:13.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:01:13.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-545460846597 2024-03-30T12:01:13.262 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:13.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=545460846597 2024-03-30T12:01:13.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 545460846597' 2024-03-30T12:01:13.263 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 545460846597 2024-03-30T12:01:13.263 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:01:13.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 545460846597 -lt 545460846597 2024-03-30T12:01:13.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:01:13.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:13.546 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:01:13.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:01:14.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:01:14.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:01:14.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:14.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:01:14.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:01:14.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:01:14.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:01:14.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-30T12:01:14.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:01:14.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:01:14.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:01:14.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:01:14.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:01:14.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:01:16.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:01:16.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:01:16.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:01:16.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:01:16.582 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:16.589+0000 7f3ca660c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:16.584 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:16.592+0000 7f3ca660c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:16.586 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:16.593+0000 7f3ca660c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:16.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:01:16.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:16.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:17.110 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:17.118+0000 7f3ca660c640 -1 Falling back to public interface 2024-03-30T12:01:17.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:18.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:18.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:18.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:01:18.150 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:01:18.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:18.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:18.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:19.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:19.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:19.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:01:19.452 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:01:19.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:19.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:19.475 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:19.482+0000 7f3ca660c640 -1 osd.1 134 log_to_monitors true 2024-03-30T12:01:19.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:01:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:01:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:20.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/4278884778,v1:127.0.0.1:6811/4278884778] [v2:127.0.0.1:6812/4278884778,v1:127.0.0.1:6813/4278884778] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:01:21.172 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:01:21.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:01:21.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:01:21.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:01:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:01:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:01:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:01:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:01:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:01:21.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:01:21.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:01:21.544 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:01:21.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:01:21.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:21.544 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:21.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650373 2024-03-30T12:01:21.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650373 2024-03-30T12:01:21.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373' 2024-03-30T12:01:21.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:21.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:21.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=592705486850 2024-03-30T12:01:21.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 592705486850 2024-03-30T12:01:21.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650373 1-592705486850' 2024-03-30T12:01:21.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:21.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650373 2024-03-30T12:01:21.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:21.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:01:21.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650373 2024-03-30T12:01:21.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:21.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650373 2024-03-30T12:01:21.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2024-03-30T12:01:21.758 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 571230650373 2024-03-30T12:01:21.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:22.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650372 -lt 571230650373 2024-03-30T12:01:22.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:01:23.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:01:23.038 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:23.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650373 -lt 571230650373 2024-03-30T12:01:23.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:23.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-592705486850 2024-03-30T12:01:23.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:23.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:01:23.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-592705486850 2024-03-30T12:01:23.361 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:23.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=592705486850 2024-03-30T12:01:23.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2024-03-30T12:01:23.363 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 592705486850 2024-03-30T12:01:23.363 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:01:23.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 592705486850 -lt 592705486850 2024-03-30T12:01:23.644 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:01:23.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:23.645 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:23.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:01:23.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:01:23.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:01:23.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:01:23.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:01:23.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:01:23.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:01:23.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:01:24.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:01:24.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:01:24.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:24.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:24.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:01:24.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:01:24.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:01:24.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-03-30T12:01:24.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:24.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-30T12:01:24.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:01:24.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:01:24.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:01:24.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:01:24.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-30T12:01:24.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:01:24.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:01:24.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:01:24.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:01:24.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:01:24.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-03-30T12:01:26.014 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:01:26.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:26.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:01:26.534 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:01:26.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:01:26.535 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:01:26.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:01:26.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:01:26.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:01:26.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:01:26.562 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:26.569+0000 7f8773e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:26.564 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:26.571+0000 7f8773e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:26.565 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:26.573+0000 7f8773e14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:26.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:26.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:27.079 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:27.087+0000 7f8773e14640 -1 Falling back to public interface 2024-03-30T12:01:27.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:01:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:01:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:28.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:28.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:29.388 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:29.395+0000 7f8773e14640 -1 osd.1 140 log_to_monitors true 2024-03-30T12:01:29.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:29.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:29.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:01:29.436 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:01:29.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:29.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:29.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:30.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:30.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:30.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:01:30.786 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:01:30.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:30.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:31.078 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/3181767232,v1:127.0.0.1:6811/3181767232] [v2:127.0.0.1:6812/3181767232,v1:127.0.0.1:6813/3181767232] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:01:31.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:01:31.078 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:01:31.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:01:31.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:01:31.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:01:31.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:01:31.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:01:31.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:01:31.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:01:31.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:01:31.445 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:01:31.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:01:31.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:31.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:31.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=571230650376 2024-03-30T12:01:31.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 571230650376 2024-03-30T12:01:31.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376' 2024-03-30T12:01:31.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:31.551 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:31.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-03-30T12:01:31.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-03-30T12:01:31.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-571230650376 1-618475290626' 2024-03-30T12:01:31.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:31.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-571230650376 2024-03-30T12:01:31.656 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:31.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:01:31.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-571230650376 2024-03-30T12:01:31.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:31.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=571230650376 2024-03-30T12:01:31.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 571230650376' 2024-03-30T12:01:31.659 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 571230650376 2024-03-30T12:01:31.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:31.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650375 -lt 571230650376 2024-03-30T12:01:31.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:01:32.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:01:32.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:33.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 571230650376 -lt 571230650376 2024-03-30T12:01:33.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:33.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-03-30T12:01:33.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:33.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:01:33.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-03-30T12:01:33.239 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:33.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-03-30T12:01:33.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-03-30T12:01:33.241 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 618475290626 2024-03-30T12:01:33.241 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:01:33.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-03-30T12:01:33.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:01:33.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:33.536 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:33.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:01:33.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:01:33.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:01:33.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:01:33.894 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:01:33.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:01:33.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:01:33.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:01:34.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:01:34.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:01:34.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:34.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:34.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:01:34.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:01:34.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:01:34.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:34.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-30T12:01:34.526 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-30T12:01:34.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:01:34.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:01:34.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:01:34.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-03-30T12:01:35.986 INFO:tasks.workunit.client.0.smithi003.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:01:36.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:01:36.758 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:01:36.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:01:36.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:01:36.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:01:36.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:01:36.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:01:36.760 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:01:36.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:01:36.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:01:36.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:01:36.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:01:36.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:01:36.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:01:36.786 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:36.794+0000 7fc29360c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:36.789 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:36.796+0000 7fc29360c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:36.790 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:36.798+0000 7fc29360c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:37.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:01:37.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:37.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:37.302 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:37.310+0000 7fc29360c640 -1 Falling back to public interface 2024-03-30T12:01:37.336 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:38.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:38.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:38.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:01:38.339 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:01:38.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:38.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:38.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:39.586 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:39.593+0000 7fc29360c640 -1 osd.0 145 log_to_monitors true 2024-03-30T12:01:39.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:39.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:39.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:01:39.632 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:01:39.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:39.633 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 149 up_thru 144 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/3433717720,v1:127.0.0.1:6803/3433717720] [v2:127.0.0.1:6804/3433717720,v1:127.0.0.1:6805/3433717720] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:01:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:01:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:01:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:01:39.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:01:39.970 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:01:40.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:01:40.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:01:40.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:01:40.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:01:40.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:01:40.058 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:01:40.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:01:40.617 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:01:40.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:01:40.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:40.617 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:40.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-03-30T12:01:40.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-03-30T12:01:40.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-03-30T12:01:40.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:40.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:40.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-03-30T12:01:40.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-03-30T12:01:40.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-03-30T12:01:40.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:40.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-03-30T12:01:40.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:40.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:01:40.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-03-30T12:01:40.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:40.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-03-30T12:01:40.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-03-30T12:01:40.867 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 639950127106 2024-03-30T12:01:40.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:41.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 639950127106 2024-03-30T12:01:41.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:01:42.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:01:42.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:42.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-03-30T12:01:42.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:42.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-03-30T12:01:42.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:42.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:01:42.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-03-30T12:01:42.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:42.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-03-30T12:01:42.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-03-30T12:01:42.476 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 618475290629 2024-03-30T12:01:42.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:01:42.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290629 -lt 618475290629 2024-03-30T12:01:42.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:01:42.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:42.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:01:43.115 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:01:43.116 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:01:43.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:01:43.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:01:43.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:43.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:01:43.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:01:43.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:01:44.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:01:44.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-30T12:01:45.320 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-30T12:01:45.320 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:01:45.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:01:45.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:01:45.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:01:45.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:01:45.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:01:45.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:01:45.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:01:45.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:01:45.838 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:01:45.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:01:45.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:01:45.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:01:45.839 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:01:45.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:45.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:01:45.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:01:45.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:01:45.841 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:01:45.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:01:45.842 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:01:45.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:01:45.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:01:45.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:01:45.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:01:45.866 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:45.874+0000 7f8384268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:45.869 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:45.876+0000 7f8384268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:45.870 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:45.878+0000 7f8384268640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:46.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:46.385 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:46.393+0000 7f8384268640 -1 Falling back to public interface 2024-03-30T12:01:46.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:47.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:47.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:47.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:01:47.426 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:01:47.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:47.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:47.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:48.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:48.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:48.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:01:48.718 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:01:48.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:48.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:48.759 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:48.767+0000 7f8384268640 -1 osd.1 150 log_to_monitors true 2024-03-30T12:01:49.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:50.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:50.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:50.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:01:50.037 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:01:50.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:50.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/854772807,v1:127.0.0.1:6811/854772807] [v2:127.0.0.1:6812/854772807,v1:127.0.0.1:6813/854772807] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:01:50.412 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:01:50.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:01:50.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:01:50.413 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:01:50.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:01:50.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:01:50.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:01:50.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:01:50.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:01:50.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:50.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:01:50.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127109 2024-03-30T12:01:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127109 2024-03-30T12:01:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109' 2024-03-30T12:01:50.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:01:50.885 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:01:50.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-03-30T12:01:50.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-03-30T12:01:50.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-03-30T12:01:50.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:50.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127109 2024-03-30T12:01:50.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:50.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:01:50.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127109 2024-03-30T12:01:50.991 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127109 2024-03-30T12:01:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-03-30T12:01:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 639950127109 2024-03-30T12:01:50.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:51.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127109 2024-03-30T12:01:51.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:01:52.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:01:52.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:01:52.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127109 -lt 639950127109 2024-03-30T12:01:52.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:01:52.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-03-30T12:01:52.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:01:52.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:01:52.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-03-30T12:01:52.584 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:01:52.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-03-30T12:01:52.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-03-30T12:01:52.585 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 661424963586 2024-03-30T12:01:52.586 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:01:52.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-03-30T12:01:52.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:01:52.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:52.865 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:53.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:01:53.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:01:53.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:01:53.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:01:53.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:01:53.514 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:01:53.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:01:53.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:01:53.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:01:53.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:01:53.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-30T12:01:53.864 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:01:53.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:01:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:01:54.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:01:55.441 INFO:tasks.workunit.client.0.smithi003.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-30T12:01:56.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:01:56.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:01:56.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:01:56.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:01:56.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:01:56.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:01:56.031 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:01:56.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:01:56.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:01:56.033 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:01:56.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:01:56.033 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:01:56.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:01:56.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:01:56.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:01:56.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:01:56.060 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:56.067+0000 7f4e8fb71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:56.062 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:56.070+0000 7f4e8fb71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:56.063 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:56.071+0000 7f4e8fb71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:01:56.326 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:01:56.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:56.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:56.581 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:56.589+0000 7f4e8fb71640 -1 Falling back to public interface 2024-03-30T12:01:56.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:57.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:57.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:57.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:01:57.607 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:01:57.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:57.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:57.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:01:58.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:01:58.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:01:58.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:01:58.901 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:01:58.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:01:58.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:01:58.928 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:01:58.935+0000 7f4e8fb71640 -1 osd.0 156 log_to_monitors true 2024-03-30T12:01:59.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:00.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:00.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:00.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:02:00.243 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:02:00.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:00.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:02:00.668 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/331723559,v1:127.0.0.1:6803/331723559] [v2:127.0.0.1:6804/331723559,v1:127.0.0.1:6805/331723559] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:02:00.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:02:00.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:02:00.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:02:00.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:02:00.669 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:02:00.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:02:00.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:02:00.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:02:00.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:02:00.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:02:00.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:02:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:02:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:02:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:02:01.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:01.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:02:01.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-03-30T12:02:01.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-03-30T12:02:01.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-03-30T12:02:01.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:01.227 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:02:01.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-03-30T12:02:01.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-03-30T12:02:01.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-03-30T12:02:01.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:01.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-03-30T12:02:01.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:01.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:02:01.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-03-30T12:02:01.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:01.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-03-30T12:02:01.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-03-30T12:02:01.360 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 687194767362 2024-03-30T12:02:01.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:01.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 687194767362 2024-03-30T12:02:01.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:02:02.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:02:02.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:02.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-03-30T12:02:02.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:02.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-03-30T12:02:02.934 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:02.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:02:02.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-03-30T12:02:02.936 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:02.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-03-30T12:02:02.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-03-30T12:02:02.937 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 661424963589 2024-03-30T12:02:02.937 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:02:03.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-03-30T12:02:03.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:02:03.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:03.235 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:03.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:03.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:02:03.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:03.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:03.867 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:04.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:02:04.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:02:04.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:02:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:02:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:02:04.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:02:04.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-30T12:02:05.440 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-30T12:02:05.440 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:02:05.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:02:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:02:05.985 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:02:05.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:02:05.986 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:02:05.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:02:05.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:02:05.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:02:05.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:02:06.012 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:06.020+0000 7fd42178a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:06.015 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:06.023+0000 7fd42178a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:06.016 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:06.024+0000 7fd42178a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:06.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:02:06.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:02:06.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:06.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:06.537 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:06.545+0000 7fd42178a640 -1 Falling back to public interface 2024-03-30T12:02:06.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:07.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:07.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:07.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:02:07.563 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:02:07.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:07.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:07.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:08.650 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:08.658+0000 7fd42178a640 -1 osd.1 161 log_to_monitors true 2024-03-30T12:02:08.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:02:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:02:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:08.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 165 up_thru 160 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/2612421950,v1:127.0.0.1:6811/2612421950] [v2:127.0.0.1:6812/2612421950,v1:127.0.0.1:6813/2612421950] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:02:09.179 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:02:09.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:02:09.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:02:09.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:02:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:02:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:02:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:02:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:02:09.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:02:09.268 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:02:09.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:02:09.623 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:02:09.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:02:09.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:09.623 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:02:09.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767365 2024-03-30T12:02:09.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767365 2024-03-30T12:02:09.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365' 2024-03-30T12:02:09.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:09.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:02:09.975 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603842 2024-03-30T12:02:09.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603842 2024-03-30T12:02:09.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365 1-708669603842' 2024-03-30T12:02:09.976 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:09.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767365 2024-03-30T12:02:09.976 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:09.977 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:02:09.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767365 2024-03-30T12:02:09.978 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:09.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767365 2024-03-30T12:02:09.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2024-03-30T12:02:09.979 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 687194767365 2024-03-30T12:02:09.979 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:10.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767365 2024-03-30T12:02:10.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:02:11.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:02:11.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:11.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767365 2024-03-30T12:02:11.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:11.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603842 2024-03-30T12:02:11.559 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:11.560 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:02:11.560 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603842 2024-03-30T12:02:11.560 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:11.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603842 2024-03-30T12:02:11.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-03-30T12:02:11.562 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 708669603842 2024-03-30T12:02:11.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:02:11.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603842 -lt 708669603842 2024-03-30T12:02:11.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:02:11.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:11.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:12.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:02:12.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:12.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:12.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:12.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:12.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:12.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:12.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:12.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:02:12.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:12.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:12.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:12.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:02:12.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T12:02:12.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T12:02:12.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T12:02:12.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:12.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:12.941 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:13.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:02:13.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:13.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:13.218 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:02:13.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:02:13.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:02:13.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:02:13.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T12:02:13.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:13.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T12:02:14.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:14.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:14.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:14.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:14.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:14.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:14.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:14.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:14.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:02:14.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:14.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:14.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:02:14.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:02:14.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:02:14.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:02:14.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:02:15.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:02:15.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:02:15.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T12:02:15.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:15.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T12:02:15.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:15.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:15.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:15.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:15.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:15.524 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:15.525 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:15.525 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:15.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:02:15.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:15.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:15.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:16.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:02:16.193 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:02:16.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:02:16.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:02:16.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-30T12:02:16.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:16.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-30T12:02:17.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:17.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:17.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:17.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:17.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:17.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:17.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:17.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:17.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:02:17.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:17.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:17.640 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:17.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:02:17.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:02:17.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:02:17.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:02:17.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-30T12:02:17.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-03-30T12:02:17.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-30T12:02:17.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:02:17.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:02:17.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:02:18.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:02:18.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:18.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:02:18.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:02:18.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:02:18.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:02:18.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:02:18.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:19.504 INFO:tasks.workunit.client.0.smithi003.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-30T12:02:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:02:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:02:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:02:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:02:20.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:02:20.102 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:02:20.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:02:20.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:02:20.104 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:02:20.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:02:20.105 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:02:20.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:02:20.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:02:20.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:02:20.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:02:20.131 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:20.139+0000 7f349c2ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:20.133 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:20.141+0000 7f349c2ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:20.135 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:20.143+0000 7f349c2ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:20.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:02:20.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:02:20.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:02:20.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:02:20.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:02:20.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:20.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:02:20.403 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:02:20.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:20.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:02:20.653 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:20.660+0000 7f349c2ae640 -1 Falling back to public interface 2024-03-30T12:02:20.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:21.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:21.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:21.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:02:21.688 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:02:21.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:21.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:02:21.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:22.979 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:22.986+0000 7f349c2ae640 -1 osd.0 167 log_to_monitors true 2024-03-30T12:02:22.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:22.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:22.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:02:22.985 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:02:22.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:22.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:02:23.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:24.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:24.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:24.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:02:24.401 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:02:24.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:24.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:02:24.707 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/992568242,v1:127.0.0.1:6803/992568242] [v2:127.0.0.1:6804/992568242,v1:127.0.0.1:6805/992568242] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:02:24.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:02:24.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:02:24.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:02:24.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:02:24.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:02:24.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:02:24.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:02:24.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:02:24.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:02:24.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:02:25.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:02:25.077 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:02:25.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:02:25.077 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:25.077 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:02:25.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407618 2024-03-30T12:02:25.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407618 2024-03-30T12:02:25.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618' 2024-03-30T12:02:25.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:25.207 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:02:25.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603846 2024-03-30T12:02:25.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603846 2024-03-30T12:02:25.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618 1-708669603846' 2024-03-30T12:02:25.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:25.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407618 2024-03-30T12:02:25.312 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:25.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:02:25.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407618 2024-03-30T12:02:25.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:25.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407618 2024-03-30T12:02:25.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-03-30T12:02:25.315 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 734439407618 2024-03-30T12:02:25.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:25.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-03-30T12:02:25.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:02:26.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:02:26.597 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:26.888 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407618 -lt 734439407618 2024-03-30T12:02:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:26.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603846 2024-03-30T12:02:26.889 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:26.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:02:26.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603846 2024-03-30T12:02:26.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603846 2024-03-30T12:02:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603846' 2024-03-30T12:02:26.891 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 708669603846 2024-03-30T12:02:26.892 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:02:27.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603846 -lt 708669603846 2024-03-30T12:02:27.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:02:27.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:27.179 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:27.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:02:27.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:27.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:27.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:27.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:27.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:27.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:27.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:27.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:02:27.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:27.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:27.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:28.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:02:28.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:02:28.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:02:28.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:02:28.184 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:02:28.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:02:28.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:02:28.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:02:28.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:02:28.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:28.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:02:28.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:02:28.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:02:28.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:02:28.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:02:28.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-30T12:02:29.499 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-30T12:02:29.499 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:02:30.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:02:30.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:02:30.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:02:30.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:02:30.039 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:02:30.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:02:30.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:02:30.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:02:30.068 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:30.075+0000 7fad6ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:30.070 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:30.078+0000 7fad6ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:30.071 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:30.079+0000 7fad6ca0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:30.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:02:30.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:02:30.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:02:30.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:02:30.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:02:30.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:30.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:02:30.338 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:02:30.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:30.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:30.587 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:30.595+0000 7fad6ca0d640 -1 Falling back to public interface 2024-03-30T12:02:30.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:31.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:31.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:31.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:02:31.623 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:02:31.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:31.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:31.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:32.723 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:32.731+0000 7fad6ca0d640 -1 osd.1 173 log_to_monitors true 2024-03-30T12:02:32.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:32.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:32.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:02:32.915 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:02:32.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:32.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 177 up_thru 171 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/2166700166,v1:127.0.0.1:6811/2166700166] [v2:127.0.0.1:6812/2166700166,v1:127.0.0.1:6813/2166700166] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:02:33.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:02:33.292 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:02:33.292 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:02:33.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:02:33.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:02:33.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:02:33.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:02:33.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:02:33.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:02:33.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:02:33.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:02:33.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:02:33.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:02:33.813 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:02:33.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:02:33.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:33.814 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:02:33.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407621 2024-03-30T12:02:33.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407621 2024-03-30T12:02:33.943 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621' 2024-03-30T12:02:33.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:33.944 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:02:34.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=760209211394 2024-03-30T12:02:34.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 760209211394 2024-03-30T12:02:34.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621 1-760209211394' 2024-03-30T12:02:34.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:34.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407621 2024-03-30T12:02:34.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:34.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:02:34.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407621 2024-03-30T12:02:34.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:34.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407621 2024-03-30T12:02:34.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407621' 2024-03-30T12:02:34.063 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 734439407621 2024-03-30T12:02:34.063 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:34.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407621 -lt 734439407621 2024-03-30T12:02:34.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:34.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-760209211394 2024-03-30T12:02:34.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:34.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:02:34.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-760209211394 2024-03-30T12:02:34.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:34.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=760209211394 2024-03-30T12:02:34.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-03-30T12:02:34.348 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 760209211394 2024-03-30T12:02:34.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:02:34.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 760209211394 -lt 760209211394 2024-03-30T12:02:34.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:02:34.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:34.632 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:34.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:02:34.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:34.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:34.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:34.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:34.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:34.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:34.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:35.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:02:35.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:35.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:35.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:35.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-30T12:02:35.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-30T12:02:35.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T12:02:35.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-30T12:02:35.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:35.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T12:02:35.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:35.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:35.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:02:35.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:35.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:35.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:02:36.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:02:36.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:02:36.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:02:36.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T12:02:36.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:36.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T12:02:36.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:36.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:36.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:36.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:36.902 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:36.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:36.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:36.903 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:37.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:02:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:37.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:37.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:02:37.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:02:37.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:02:37.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:02:37.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T12:02:37.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:37.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T12:02:38.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:38.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:38.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:38.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:38.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:38.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:38.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:38.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:38.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:02:38.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:38.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:38.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:02:38.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:02:38.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:02:38.916 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:02:39.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:02:39.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:02:39.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-30T12:02:39.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:02:39.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-30T12:02:40.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:02:40.070 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:40.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:40.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:40.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:40.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:40.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:40.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:40.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:02:40.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:40.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:40.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:02:40.717 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:02:40.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:02:41.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:02:41.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-30T12:02:41.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:02:41.023 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:02:41.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:02:41.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:02:41.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:02:41.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-03-30T12:02:41.820 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-30T12:02:41.821 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:02:42.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:02:42.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:02:42.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:02:42.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:02:42.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:02:42.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:02:42.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:02:42.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:02:42.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:02:42.368 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:42.375+0000 7ff06ba11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:42.370 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:42.378+0000 7ff06ba11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:42.371 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:42.379+0000 7ff06ba11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:42.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:02:42.635 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:02:42.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:42.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:42.885 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:42.893+0000 7ff06ba11640 -1 Falling back to public interface 2024-03-30T12:02:42.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:43.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:43.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:43.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:02:43.917 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:02:43.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:43.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:44.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:02:45.015 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:45.023+0000 7ff06ba11640 -1 osd.1 179 log_to_monitors true 2024-03-30T12:02:45.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:02:45.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:45.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:02:45.213 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:02:45.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:45.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:02:45.612 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 183 up_thru 177 down_at 180 last_clean_interval [177,179) [v2:127.0.0.1:6810/4291936028,v1:127.0.0.1:6811/4291936028] [v2:127.0.0.1:6812/4291936028,v1:127.0.0.1:6813/4291936028] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:02:45.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:02:45.614 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:02:45.614 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:02:45.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:02:45.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:02:45.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:02:45.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:02:45.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:02:45.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:02:45.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:02:45.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:02:45.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:02:46.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:02:46.182 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:02:46.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:02:46.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:46.182 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:02:46.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407624 2024-03-30T12:02:46.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407624 2024-03-30T12:02:46.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624' 2024-03-30T12:02:46.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:02:46.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:02:46.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015170 2024-03-30T12:02:46.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015170 2024-03-30T12:02:46.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624 1-785979015170' 2024-03-30T12:02:46.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:46.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407624 2024-03-30T12:02:46.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:46.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:02:46.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407624 2024-03-30T12:02:46.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:46.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407624 2024-03-30T12:02:46.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2024-03-30T12:02:46.530 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 734439407624 2024-03-30T12:02:46.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:46.813 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407623 -lt 734439407624 2024-03-30T12:02:46.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:02:47.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:02:47.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:48.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407623 -lt 734439407624 2024-03-30T12:02:48.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:02:49.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T12:02:49.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:02:49.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407624 -lt 734439407624 2024-03-30T12:02:49.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:02:49.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015170 2024-03-30T12:02:49.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:02:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:02:49.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015170 2024-03-30T12:02:49.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:02:49.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015170 2024-03-30T12:02:49.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-03-30T12:02:49.404 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 785979015170 2024-03-30T12:02:49.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:02:49.728 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015170 -lt 785979015170 2024-03-30T12:02:49.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:02:49.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:49.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:02:50.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:02:50.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:02:50.364 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:02:50.364 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:02:50.364 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:02:50.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:02:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:02:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:02:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-30T12:02:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-30T12:02:50.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-03-30T12:02:50.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-30T12:02:50.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-30T12:02:50.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-30T12:02:50.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-30T12:02:50.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-30T12:02:50.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-30T12:02:50.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:02:50.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-30T12:02:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-03-30T12:02:51.095 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0: {} 2024-03-30T12:02:51.096 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-30T12:02:51.112 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1: {} 2024-03-30T12:02:51.113 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-30T12:02:51.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-30T12:02:51.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:02:51.123 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-30T12:02:51.124 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-30T12:02:51.124 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:02:51.422 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:02:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:02:51.422 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:02:51.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:02:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:02:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-30T12:02:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:02:51.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:02:51.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-30T12:02:51.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:02:51.425 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:02:51.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:02:51.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:02:51.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:02:51.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:02:51.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-30T12:02:51.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-30T12:02:51.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:02:51.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-30T12:02:51.608 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-30T12:02:51.608 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:02:51.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:02:51.890 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:02:51.890 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:02:51.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:02:51.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:02:51.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-30T12:02:51.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:02:51.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:02:51.891 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-30T12:02:51.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:02:51.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:02:51.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:02:51.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:51.983 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:51.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:02:51.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:02:51.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-30T12:02:52.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-30T12:02:52.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:02:52.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-30T12:02:52.074 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-30T12:02:52.075 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:02:52.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-30T12:02:52.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:02:52.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-30T12:02:52.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:02:52.359 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:02:52.360 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:02:52.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:02:52.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:02:52.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:02:52.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-30T12:02:52.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:02:52.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:02:52.449 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:52.450 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:52.450 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:52.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:02:52.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:02:52.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-30T12:02:52.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-30T12:02:52.541 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:02:52.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:02:52.823 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:02:52.823 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:02:52.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:02:52.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:02:52.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-30T12:02:52.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:02:52.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:02:52.824 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:52.826 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:52.827 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:02:52.827 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:02:52.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:02:52.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:02:52.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-30T12:02:53.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-03-30T12:02:53.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-30T12:02:53.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-30T12:02:53.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:02:53.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:02:53.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:02:53.005 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:02:53.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-30T11:58:57.119155+0000 2024-03-30T12:02:53.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-30T12:02:53.550 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-30T11:58:57.119155+0000 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:58:57.119155+0000 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:02:53.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:02:53.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:57.119155+0000 '>' 2024-03-30T11:58:57.119155+0000 2024-03-30T12:02:53.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:02:54.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:02:54.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:02:54.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:02:54.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:02:54.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:02:54.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:02:54.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:02:55.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:57.119155+0000 '>' 2024-03-30T11:58:57.119155+0000 2024-03-30T12:02:55.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:02:56.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:02:56.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:02:56.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:02:56.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:02:56.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:02:56.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:02:56.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:02:56.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:02:53.986125+0000 '>' 2024-03-30T11:58:57.119155+0000 2024-03-30T12:02:56.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:02:56.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-03-30T12:02:56.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-03-30T12:02:56.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-30T12:02:56.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-30T12:02:56.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.497 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.505 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:02:56.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-03-30T12:02:56.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-30T12:02:56.650 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-30T12:02:56.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-30T12:02:56.652 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T12:02:56.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-30T12:02:56.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-30T12:02:56.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-30T12:02:56.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=183 2024-03-30T12:02:56.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-30T12:02:56.684 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T12:02:56.684 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T12:02:56.684 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T12:02:56.684 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T12:02:56.685 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T12:02:56.686 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T12:02:56.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-30T12:02:56.686 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T12:02:56.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-30T12:02:56.708 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T12:02:56.708 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T12:02:56.708 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T12:02:56.708 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-30T12:02:56.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T12:02:56.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T12:02:56.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T12:02:56.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-03-30T12:02:56.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-30T12:02:56.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-30T12:02:56.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-03-30T12:02:56.732 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-30T12:02:56.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:02:56.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:02:56.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:02:56.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:02:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:02:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:02:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:02:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:02:57.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:02:57.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:02:57.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:02:57.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-30T12:02:58.073 INFO:tasks.workunit.client.0.smithi003.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-30T12:02:58.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:02:58.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:02:58.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:02:58.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:02:58.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:02:58.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:58.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:02:58.595 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:02:58.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:02:58.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:02:58.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:02:58.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:02:58.620 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:58.628+0000 7ffb2200d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:58.622 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:58.630+0000 7ffb2200d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:58.624 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:58.631+0000 7ffb2200d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:02:58.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:02:59.138 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:02:59.146+0000 7ffb2200d640 -1 Falling back to public interface 2024-03-30T12:02:59.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:00.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:00.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:00.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:03:00.190 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:03:00.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:00.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:00.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:01.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:01.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:01.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:03:01.482 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:03:01.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:01.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:01.576 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:01.584+0000 7ffb2200d640 -1 osd.0 185 log_to_monitors true 2024-03-30T12:03:01.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:02.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:02.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:02.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:03:02.779 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:03:02.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:02.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 189 up_thru 189 down_at 186 last_clean_interval [171,185) [v2:127.0.0.1:6802/1244182289,v1:127.0.0.1:6803/1244182289] [v2:127.0.0.1:6804/1244182289,v1:127.0.0.1:6805/1244182289] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:03:03.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:03:03.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:03:03.150 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:03:03.150 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:03:03.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:03:03.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:03:03.150 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:03:03.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:03:03.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:03:03.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:03:03.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:03:03.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:03:03.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:03:03.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:03:03.518 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:03:03.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:03:03.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:03.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:03:03.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818946 2024-03-30T12:03:03.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818946 2024-03-30T12:03:03.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946' 2024-03-30T12:03:03.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:03.679 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:03:03.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015174 2024-03-30T12:03:03.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015174 2024-03-30T12:03:03.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946 1-785979015174' 2024-03-30T12:03:03.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:03.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818946 2024-03-30T12:03:03.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:03.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:03:03.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818946 2024-03-30T12:03:03.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:03.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818946 2024-03-30T12:03:03.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-03-30T12:03:03.767 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 811748818946 2024-03-30T12:03:03.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:04.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 811748818946 2024-03-30T12:03:04.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:03:05.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:03:05.048 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:05.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818946 -lt 811748818946 2024-03-30T12:03:05.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:05.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015174 2024-03-30T12:03:05.383 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:05.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:03:05.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015174 2024-03-30T12:03:05.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:05.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015174 2024-03-30T12:03:05.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015174' 2024-03-30T12:03:05.386 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 785979015174 2024-03-30T12:03:05.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:05.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015174 -lt 785979015174 2024-03-30T12:03:05.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:03:05.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:05.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:06.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:03:06.024 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:06.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:06.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:06.024 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:06.025 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:06.025 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:06.025 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:06.308 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:03:06.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:06.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:06.309 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:06.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:03:06.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:03:06.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:03:06.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-30T12:03:06.663 INFO:tasks.workunit.client.0.smithi003.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-30T12:03:06.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-30T12:03:06.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-30T12:03:06.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:03:06.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:03:06.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:03:06.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:03:06.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:03:06.665 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:03:06.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:03:06.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:03:06.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:03:06.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:03:06.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-30T12:03:06.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:03:06.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:03:06.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:03:06.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:03:06.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:03:06.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-03-30T12:03:07.819 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-30T12:03:07.819 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:03:08.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:03:08.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:03:08.337 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:03:08.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:03:08.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:03:08.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:03:08.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:03:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:03:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:03:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:03:08.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:03:08.367 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:08.374+0000 7f68f94bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:08.369 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:08.376+0000 7f68f94bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:08.370 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:08.378+0000 7f68f94bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:08.630 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:08.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:03:08.884 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:08.892+0000 7f68f94bd640 -1 Falling back to public interface 2024-03-30T12:03:08.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:09.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:09.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:09.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:03:09.957 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:03:09.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:09.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:03:10.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:10.913 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:10.921+0000 7f68f94bd640 -1 osd.1 191 log_to_monitors true 2024-03-30T12:03:11.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:11.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:11.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:03:11.259 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:03:11.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:11.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:03:11.612 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 195 up_thru 195 down_at 192 last_clean_interval [183,191) [v2:127.0.0.1:6810/1496740439,v1:127.0.0.1:6811/1496740439] [v2:127.0.0.1:6812/1496740439,v1:127.0.0.1:6813/1496740439] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:03:11.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:03:11.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:03:11.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:03:11.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:03:11.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:03:11.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:03:11.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:03:11.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:03:11.700 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:03:12.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:03:12.041 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:03:12.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:03:12.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:12.041 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:03:12.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818949 2024-03-30T12:03:12.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818949 2024-03-30T12:03:12.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949' 2024-03-30T12:03:12.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:12.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:03:12.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622722 2024-03-30T12:03:12.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622722 2024-03-30T12:03:12.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818949 1-837518622722' 2024-03-30T12:03:12.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:12.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818949 2024-03-30T12:03:12.409 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:12.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:03:12.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818949 2024-03-30T12:03:12.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:12.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818949 2024-03-30T12:03:12.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818949' 2024-03-30T12:03:12.412 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 811748818949 2024-03-30T12:03:12.412 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:12.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818949 -lt 811748818949 2024-03-30T12:03:12.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:12.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622722 2024-03-30T12:03:12.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:12.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:03:12.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622722 2024-03-30T12:03:12.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:12.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622722 2024-03-30T12:03:12.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622722' 2024-03-30T12:03:12.701 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 837518622722 2024-03-30T12:03:12.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:12.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 837518622722 2024-03-30T12:03:12.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:03:13.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:03:13.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:14.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 837518622722 2024-03-30T12:03:14.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:03:15.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T12:03:15.284 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:15.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622722 -lt 837518622722 2024-03-30T12:03:15.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:03:15.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:15.576 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:15.926 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:16.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:03:16.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:16.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:16.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:03:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T12:03:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T12:03:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T12:03:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:03:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T12:03:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:03:16.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:16.658 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:16.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:03:16.938 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:16.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:16.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:17.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:03:17.290 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:03:17.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:03:17.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:03:17.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T12:03:17.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:03:17.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:17.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:18.154 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:03:18.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:18.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:18.154 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:18.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:03:18.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:03:18.889 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:03:18.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:03:18.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T12:03:18.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:03:18.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:19.291 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:19.292 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:19.569 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:03:19.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:19.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:19.570 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:19.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:03:19.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:03:19.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:03:19.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:03:20.025 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:03:20.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:03:20.026 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:03:20.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:03:20.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:03:20.026 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:03:20.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:03:20.333 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:21.519 INFO:tasks.workunit.client.0.smithi003.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-30T12:03:22.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:03:22.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:03:22.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:03:22.044 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:03:22.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:03:22.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:03:22.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:03:22.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:03:22.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:03:22.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:03:22.070 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:22.078+0000 7feafebac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:22.072 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:22.080+0000 7feafebac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:22.073 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:22.081+0000 7feafebac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:22.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:03:22.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:03:22.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:03:22.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:03:22.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:03:22.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:22.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:03:22.342 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:03:22.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:22.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:22.595+0000 7feafebac640 -1 Falling back to public interface 2024-03-30T12:03:22.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:23.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:23.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:23.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:03:23.623 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:03:23.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:23.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:23.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:24.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:24.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:24.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:03:24.919 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:03:24.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:24.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:24.951 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:24.959+0000 7feafebac640 -1 osd.0 197 log_to_monitors true 2024-03-30T12:03:25.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:26.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:26.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:26.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:03:26.316 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:03:26.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:26.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:03:26.663 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 201 up_thru 201 down_at 198 last_clean_interval [189,197) [v2:127.0.0.1:6802/522750630,v1:127.0.0.1:6803/522750630] [v2:127.0.0.1:6804/522750630,v1:127.0.0.1:6805/522750630] exists,up d40cc1e9-2219-4a47-83ab-fe6e8ceea4c0 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:03:26.664 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:03:26.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:03:26.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:03:26.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:03:26.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:03:26.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:03:26.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:03:27.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:03:27.033 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:03:27.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:03:27.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:27.033 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:03:27.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426498 2024-03-30T12:03:27.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426498 2024-03-30T12:03:27.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498' 2024-03-30T12:03:27.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:27.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:03:27.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622726 2024-03-30T12:03:27.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622726 2024-03-30T12:03:27.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498 1-837518622726' 2024-03-30T12:03:27.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:27.243 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426498 2024-03-30T12:03:27.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:27.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:03:27.245 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426498 2024-03-30T12:03:27.245 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:27.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426498 2024-03-30T12:03:27.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426498' 2024-03-30T12:03:27.246 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 863288426498 2024-03-30T12:03:27.247 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:27.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 863288426498 2024-03-30T12:03:27.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:03:28.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:03:28.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:28.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426498 -lt 863288426498 2024-03-30T12:03:28.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:28.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622726 2024-03-30T12:03:28.826 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:28.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:03:28.827 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622726 2024-03-30T12:03:28.827 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:28.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622726 2024-03-30T12:03:28.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622726' 2024-03-30T12:03:28.829 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 837518622726 2024-03-30T12:03:28.829 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:29.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622726 -lt 837518622726 2024-03-30T12:03:29.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:03:29.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:29.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:29.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:03:29.481 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:29.481 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:29.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:29.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:29.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:29.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:29.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:29.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:03:29.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:29.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:29.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:30.135 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:03:30.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:03:30.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:03:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:03:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:03:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:03:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:03:30.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:03:30.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:03:30.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-30T12:03:31.279 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-30T12:03:31.279 INFO:tasks.workunit.client.0.smithi003.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:03:31.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:03:31.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:03:31.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:03:31.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:03:31.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:03:31.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:03:31.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:03:31.802 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:03:31.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:03:31.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:03:31.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:03:31.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:03:31.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:03:31.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:03:31.829 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:31.837+0000 7fc08e2a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:31.832 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:31.839+0000 7fc08e2a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:31.833 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:31.841+0000 7fc08e2a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:32.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:03:32.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:03:32.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:03:32.095 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:03:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:03:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:03:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:03:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:32.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:03:32.350 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:32.358+0000 7fc08e2a8640 -1 Falling back to public interface 2024-03-30T12:03:32.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:33.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:33.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:33.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:03:33.378 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:03:33.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:33.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:03:33.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:03:34.579 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:34.585+0000 7fc08e2a8640 -1 osd.1 203 log_to_monitors true 2024-03-30T12:03:34.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:03:34.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:03:34.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:03:34.716 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:03:34.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:03:34.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 207 up_thru 201 down_at 204 last_clean_interval [195,203) [v2:127.0.0.1:6810/4038566154,v1:127.0.0.1:6811/4038566154] [v2:127.0.0.1:6812/4038566154,v1:127.0.0.1:6813/4038566154] exists,up 0708f773-8546-4cd5-8339-50541a603197 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:03:35.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:03:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:03:35.036 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:03:35.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:03:35.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:03:35.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:03:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:03:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:03:35.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:03:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:03:35.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:03:35.124 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:03:35.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:03:35.427 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:03:35.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:03:35.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:35.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:03:35.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426501 2024-03-30T12:03:35.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426501 2024-03-30T12:03:35.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501' 2024-03-30T12:03:35.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:35.534 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:03:35.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230274 2024-03-30T12:03:35.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230274 2024-03-30T12:03:35.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501 1-889058230274' 2024-03-30T12:03:35.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:35.710 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426501 2024-03-30T12:03:35.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:35.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:03:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426501 2024-03-30T12:03:35.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:35.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426501 2024-03-30T12:03:35.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426501' 2024-03-30T12:03:35.713 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 863288426501 2024-03-30T12:03:35.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:36.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426499 -lt 863288426501 2024-03-30T12:03:36.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:03:37.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:03:37.046 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:37.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426501 -lt 863288426501 2024-03-30T12:03:37.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:37.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230274 2024-03-30T12:03:37.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:37.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:03:37.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230274 2024-03-30T12:03:37.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:37.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230274 2024-03-30T12:03:37.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230274' 2024-03-30T12:03:37.342 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 889058230274 2024-03-30T12:03:37.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:37.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230274 -lt 889058230274 2024-03-30T12:03:37.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:03:37.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:37.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:37.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:03:37.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:37.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:37.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:37.981 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:37.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:37.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:37.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:38.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:03:38.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:38.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:38.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:38.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:03:38.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-30T12:03:38.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-30T12:03:38.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-30T12:03:38.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:03:38.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:38.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:38.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:03:38.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:38.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:38.990 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:03:39.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:03:39.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:03:39.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:03:39.693 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:03:39.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-30T12:03:39.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:03:39.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-30T12:03:39.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:03:39.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:39.895 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:39.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:39.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:39.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:39.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:39.896 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:40.172 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-30T12:03:40.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:40.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:40.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:40.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-30T12:03:40.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-30T12:03:40.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-30T12:03:40.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-30T12:03:40.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-30T12:03:40.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-30T12:03:40.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-30T12:03:40.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-30T12:03:40.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-30T12:03:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-30T12:03:40.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-30T12:03:40.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-30T12:03:40.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-30T12:03:40.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-30T12:03:41.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-30T12:03:41.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:41.285 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:41.285 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:41.285 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:41.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:41.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:41.286 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:41.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:03:41.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:41.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:41.564 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:41.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:03:41.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:03:41.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:03:41.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:03:41.920 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-30T12:03:41.921 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:03:42.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:03:42.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:03:42.208 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:03:42.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:03:42.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:03:42.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:03:42.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-30T12:03:42.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:03:42.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:03:42.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-30T12:03:42.388 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:03:42.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:03:42.670 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:03:42.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:03:42.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:03:42.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:03:42.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-30T12:03:42.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:03:42.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:03:42.671 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-30T12:03:42.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:03:42.673 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:03:42.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:03:42.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:03:42.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:03:42.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-30T12:03:42.762 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:03:42.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-30T12:03:42.852 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-30T12:03:42.853 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:03:42.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-30T12:03:42.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:03:42.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-30T12:03:42.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-30T12:03:42.854 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:03:43.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-30T12:03:43.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:03:43.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:03:43.140 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:03:43.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:03:43.227 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:43.229 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:43.230 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.0.asok 2024-03-30T12:03:43.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:03:43.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-30T12:03:43.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-30T12:03:43.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-30T12:03:43.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-30T12:03:43.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-30T12:03:43.321 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:03:43.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-30T12:03:43.606 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:03:43.606 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-30T12:03:43.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-30T12:03:43.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-30T12:03:43.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-30T12:03:43.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-30T12:03:43.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-30T12:03:43.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-30T12:03:43.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-30T12:03:43.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-30T12:03:43.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-30T12:03:43.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-30T12:03:43.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:03:43.610 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69973/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-30T12:03:43.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-30T12:03:43.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-30T12:03:43.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-30T12:03:43.698 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-osd.1.asok 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-30T12:03:43.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69973/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-30T12:03:43.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-03-30T12:03:43.884 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0: {} 2024-03-30T12:03:43.885 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0: osd_max_object_size = '' 2024-03-30T12:03:43.901 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1: {} 2024-03-30T12:03:43.902 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1: osd_max_object_size = '' 2024-03-30T12:03:43.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-30T12:03:43.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-30T12:03:43.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T12:03:43.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:43.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:03:43.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:43.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T12:03:44.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-30T11:58:57.119155+0000 2024-03-30T12:03:44.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-30T12:03:44.471 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-30T12:03:44.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-30T11:58:57.119155+0000 last_deep_scrub_stamp 2024-03-30T12:03:44.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T12:03:44.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T11:58:57.119155+0000 2024-03-30T12:03:44.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-30T12:03:44.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:03:44.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:03:44.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T12:03:44.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:44.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:03:44.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:44.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T12:03:44.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:57.119155+0000 '>' 2024-03-30T11:58:57.119155+0000 2024-03-30T12:03:44.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:03:45.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:03:45.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:03:45.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T12:03:45.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:45.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:03:45.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:45.769 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T12:03:46.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T11:58:57.119155+0000 '>' 2024-03-30T11:58:57.119155+0000 2024-03-30T12:03:46.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:03:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:03:47.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:03:47.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-30T12:03:47.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:47.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:03:47.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:47.060 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-30T12:03:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:03:44.617236+0000 '>' 2024-03-30T11:58:57.119155+0000 2024-03-30T12:03:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:03:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-03-30T12:03:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-30T12:03:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-03-30T12:03:47.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-03-30T12:03:47.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-30T12:03:47.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.598 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.599 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.671 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-30T12:03:47.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-03-30T12:03:47.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-30T12:03:47.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-30T12:03:47.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-30T12:03:47.766 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T12:03:47.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-30T12:03:47.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-30T12:03:47.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-30T12:03:47.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=207 2024-03-30T12:03:47.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-30T12:03:47.798 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T12:03:47.798 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T12:03:47.798 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T12:03:47.798 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T12:03:47.798 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-30T12:03:47.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T12:03:47.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T12:03:47.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T12:03:47.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-30T12:03:47.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T12:03:47.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T12:03:47.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T12:03:47.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-03-30T12:03:47.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-30T12:03:47.849 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-03-30T12:03:47.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-03-30T12:03:47.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-03-30T12:03:47.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:47.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:03:47.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:47.850 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:03:48.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T12:03:44.617236+0000 2024-03-30T12:03:48.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-03-30T12:03:48.396 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0 on osd.1 to repair 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-03-30T12:03:44.617236+0000 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:03:44.617236+0000 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:03:48.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:48.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:03:48.689 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:03:44.617236+0000 '>' 2024-03-30T12:03:44.617236+0000 2024-03-30T12:03:48.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:03:49.690 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:03:49.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:03:49.691 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:03:49.692 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:49.692 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:03:49.692 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:49.692 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:03:49.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:03:44.617236+0000 '>' 2024-03-30T12:03:44.617236+0000 2024-03-30T12:03:49.981 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:03:50.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:03:50.983 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:03:50.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:03:50.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:03:50.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:03:50.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:03:50.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:03:49.353310+0000 '>' 2024-03-30T12:03:44.617236+0000 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:03:51.275 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:03:51.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:03:51.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:03:51.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:03:51.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:03:51.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:03:51.364 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:03:51.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:03:51.643 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:03:51.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:03:51.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:51.643 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:03:52.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426505 2024-03-30T12:03:52.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426505 2024-03-30T12:03:52.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505' 2024-03-30T12:03:52.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:03:52.969 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:03:53.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230278 2024-03-30T12:03:53.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230278 2024-03-30T12:03:53.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505 1-889058230278' 2024-03-30T12:03:53.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:53.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426505 2024-03-30T12:03:53.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:53.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:03:53.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426505 2024-03-30T12:03:53.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:53.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426505 2024-03-30T12:03:53.083 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 863288426505 2024-03-30T12:03:53.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426505' 2024-03-30T12:03:53.084 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:53.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426504 -lt 863288426505 2024-03-30T12:03:53.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:03:54.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:03:54.364 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:03:54.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426505 -lt 863288426505 2024-03-30T12:03:54.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:03:54.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230278 2024-03-30T12:03:54.660 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:03:54.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:03:54.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230278 2024-03-30T12:03:54.661 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:03:54.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230278 2024-03-30T12:03:54.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230278' 2024-03-30T12:03:54.663 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 889058230278 2024-03-30T12:03:54.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:03:54.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230278 -lt 889058230278 2024-03-30T12:03:54.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:03:54.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:54.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:55.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:03:55.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:03:55.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:03:55.384 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:03:55.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:03:55.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:03:55.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:03:55.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:03:55.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:03:55.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:03:55.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:03:55.663 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:03:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:03:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:03:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:03:56.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-30T12:03:56.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-30T12:03:56.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-30T12:03:56.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-30T12:03:56.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-30T12:03:56.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-30T12:03:56.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-03-30T12:03:56.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-30T12:03:56.578 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'csr_pool' removed 2024-03-30T12:03:56.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T12:03:56.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:03:56.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:03:56.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:03:56.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:03:56.591 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:03:56.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:03:56.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:03:56.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:03:56.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:03:56.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:03:56.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:03:56.700 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:03:56.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:03:56.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:03:56.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:03:56.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:03:56.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:03:56.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:03:56.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:03:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:03:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:03:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:03:56.740 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:03:56.740 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:56.740 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:56.740 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:03:56.741 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:03:56.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:03:56.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:03:56.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:03:56.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:03:56.745 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:03:56.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:03:56.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:03:56.746 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:03:56.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:03:56.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:03:56.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:03:56.748 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:03:56.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:03:56.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:03:56.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:03:56.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:03:56.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:56.751 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:56.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:03:56.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:03:56.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:03:56.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T12:03:56.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T12:03:56.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:56.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:56.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-03-30T12:03:56.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T12:03:56.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:57.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-30T12:03:57.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T12:03:57.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T12:03:57.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:03:57.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:03:57.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T12:03:57.346 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T12:03:57.347 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:03:57.347 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:03:57.347 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:03:57.348 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:57.348 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:57.348 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:57.348 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:03:57.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:03:57.348 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T12:03:57.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T12:03:57.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:03:57.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:57.438 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:03:57.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:03:57.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T12:03:57.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-03-30T12:03:57.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T12:03:57.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T12:03:57.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T12:03:57.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T12:03:57.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T12:03:57.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:58.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:58.340 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:03:58.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:03:58.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:03:58.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:03:58.373 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:03:58.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0481bb9e-33cb-4df0-b4fd-e4765ed9a728 2024-03-30T12:03:58.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0481bb9e-33cb-4df0-b4fd-e4765ed9a728' 2024-03-30T12:03:58.374 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 0481bb9e-33cb-4df0-b4fd-e4765ed9a728 2024-03-30T12:03:58.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:03:58.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCu/wdmFOK8FxAAEDf1qmCJUThH1wK0JHxxwA== 2024-03-30T12:03:58.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCu/wdmFOK8FxAAEDf1qmCJUThH1wK0JHxxwA=="}' 2024-03-30T12:03:58.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0481bb9e-33cb-4df0-b4fd-e4765ed9a728 -i td/osd-scrub-repair/0/new.json 2024-03-30T12:03:58.602 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:03:58.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T12:03:58.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCu/wdmFOK8FxAAEDf1qmCJUThH1wK0JHxxwA== --osd-uuid 0481bb9e-33cb-4df0-b4fd-e4765ed9a728 2024-03-30T12:03:58.641 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.648+0000 7f11f47dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:58.653 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.661+0000 7f11f47dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:58.655 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.662+0000 7f11f47dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:03:58.655 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.662+0000 7f11f47dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:03:58.655 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.662+0000 7f11f47dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:03:58.655 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.662+0000 7f11f47dd640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:03:58.736 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:03:58.744+0000 7f11f47dd640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T12:04:04.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T12:04:04.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:04:04.280 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T12:04:04.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T12:04:04.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:04:04.669 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:04:04.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T12:04:04.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:04:04.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:04:04.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:04:04.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:04:04.695 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:04.703+0000 7fb029f9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:04.719 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:04.726+0000 7fb029f9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:04.719 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:04.727+0000 7fb029f9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:04.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:04:04.958 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:04:04.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:04.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:05.235 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:05.243+0000 7fb029f9a640 -1 Falling back to public interface 2024-03-30T12:04:05.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:06.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:06.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:06.241 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:04:06.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:04:06.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:06.241 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:06.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:07.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:07.536 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:07.536 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:04:07.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:04:07.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:07.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:07.575 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:07.582+0000 7fb029f9a640 -1 osd.0 0 log_to_monitors true 2024-03-30T12:04:07.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:08.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:08.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:08.834 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:04:08.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:04:08.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:08.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:09.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:10.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:10.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:10.128 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:04:10.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:04:10.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:10.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:10.565 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:11.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:11.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:11.567 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:04:11.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:04:11.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:11.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:11.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:12.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:12.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:12.860 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:04:12.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:04:12.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:12.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:13.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:14.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:14.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:14.160 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:04:14.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:04:14.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:14.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:14.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:15.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:15.541 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:04:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:04:15.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:15.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:15.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:16.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:16.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:16.836 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:04:16.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:04:16.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:16.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:17.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:18.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:18.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:18.144 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:04:18.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:04:18.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:18.145 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:18.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:19.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:19.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:19.529 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:04:19.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:04:19.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:19.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:19.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:20.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:20.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:20.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:04:20.968 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:04:20.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:20.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:21.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:22.648 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:22.649 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:04:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:04:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:22.649 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:23.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:24.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:24.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:24.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:04:24.567 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:04:24.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:24.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:24.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:25.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:25.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:25.951 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:04:25.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:04:25.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:25.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:26.610 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:27.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:27.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:27.612 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:04:27.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:04:27.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:27.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:28.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:29.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:29.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:29.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:04:29.099 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:04:29.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:29.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:29.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:30.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:30.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:30.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:04:30.397 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:04:30.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:30.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:04:30.694 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3906873828,v1:127.0.0.1:6803/3906873828] [v2:127.0.0.1:6804/3906873828,v1:127.0.0.1:6805/3906873828] exists,up 0481bb9e-33cb-4df0-b4fd-e4765ed9a728 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:04:30.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:04:30.699 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:04:30.700 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a9cd392-6867-4a93-bd1b-2bbb0199aada 2024-03-30T12:04:30.701 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 0a9cd392-6867-4a93-bd1b-2bbb0199aada 2024-03-30T12:04:30.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0a9cd392-6867-4a93-bd1b-2bbb0199aada' 2024-03-30T12:04:30.702 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:04:30.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDO/wdm9yIpKxAA19qbfU+8yM//rPE1/mUdnw== 2024-03-30T12:04:30.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDO/wdm9yIpKxAA19qbfU+8yM//rPE1/mUdnw=="}' 2024-03-30T12:04:30.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a9cd392-6867-4a93-bd1b-2bbb0199aada -i td/osd-scrub-repair/1/new.json 2024-03-30T12:04:31.050 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:04:31.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T12:04:31.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDO/wdm9yIpKxAA19qbfU+8yM//rPE1/mUdnw== --osd-uuid 0a9cd392-6867-4a93-bd1b-2bbb0199aada 2024-03-30T12:04:31.090 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.097+0000 7f4c250b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:31.134 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.142+0000 7f4c250b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:31.135 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.143+0000 7f4c250b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.143+0000 7f4c250b5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:04:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.143+0000 7f4c250b5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:04:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.143+0000 7f4c250b5640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:04:31.209 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:31.217+0000 7f4c250b5640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T12:04:37.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T12:04:37.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:04:37.225 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T12:04:37.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T12:04:37.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:04:37.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T12:04:37.593 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:04:37.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:04:37.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:04:37.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:04:37.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:04:37.619 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:37.627+0000 7f072f60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:37.621 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:37.629+0000 7f072f60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:37.622 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:37.630+0000 7f072f60f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:04:37.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T12:04:37.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:04:37.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:04:37.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:04:37.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:04:37.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:37.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:04:37.879 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:04:37.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:37.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:38.160 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:38.168+0000 7f072f60f640 -1 Falling back to public interface 2024-03-30T12:04:38.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:39.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:39.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:39.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:04:39.164 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:04:39.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:39.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:39.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:04:40.462 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:04:40.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:40.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:40.491 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:04:40.498+0000 7f072f60f640 -1 osd.1 0 log_to_monitors true 2024-03-30T12:04:40.784 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:41.786 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:41.786 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:04:41.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:04:41.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:41.787 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:42.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:43.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:43.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:43.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:04:43.139 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:04:43.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:43.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:43.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:44.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:44.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:44.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:04:44.553 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:04:44.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:44.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:44.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:45.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:45.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:45.847 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:04:45.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:04:45.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:45.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:46.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:47.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:47.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:47.144 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:04:47.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:04:47.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:47.144 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:47.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:04:48.516 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:04:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:48.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:48.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:49.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:49.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:49.829 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:04:49.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:04:49.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:49.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:50.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:51.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:51.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:51.165 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:04:51.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:04:51.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:51.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:51.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:52.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:52.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:52.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:04:52.515 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:04:52.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:52.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:52.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:04:53.867 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:04:53.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:53.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:54.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:55.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:55.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:55.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:04:55.315 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:04:55.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:55.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:55.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:56.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:56.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:56.993 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:04:56.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:04:56.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:56.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:57.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:58.325 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:58.326 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:04:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:04:58.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:58.327 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:04:58.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:04:59.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:04:59.753 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:04:59.754 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:04:59.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:04:59.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:04:59.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:00.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:01.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:01.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:01.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:05:01.366 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:05:01.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:01.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:02.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:03.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:03.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:03.067 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:05:03.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:05:03.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:03.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:03.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:04.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:04.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:04.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T12:05:04.503 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T12:05:04.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:04.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:04.840 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1730042291,v1:127.0.0.1:6811/1730042291] [v2:127.0.0.1:6812/1730042291,v1:127.0.0.1:6813/1730042291] exists,up 0a9cd392-6867-4a93-bd1b-2bbb0199aada 2024-03-30T12:05:04.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:05:04.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:05:04.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:05:04.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-03-30T12:05:04.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T12:05:05.114 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T12:05:05.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T12:05:05.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T12:05:05.581 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T12:05:05.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:05:06.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T12:05:08.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-30T12:05:08.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:05:08.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:05:08.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:05:08.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:05:08.226 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:05:08.226 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:05:08.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:05:08.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:05:08.226 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:05:08.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:05:08.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:05:08.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:05:08.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:05:08.326 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:05:08.326 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:05:08.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:05:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:05:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:05:08.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:08.608 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:05:08.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-03-30T12:05:08.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-03-30T12:05:08.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-03-30T12:05:08.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:08.743 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:05:08.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-03-30T12:05:08.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-03-30T12:05:08.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672964' 2024-03-30T12:05:08.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:08.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-03-30T12:05:08.848 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:08.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:05:08.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-03-30T12:05:08.849 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:08.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-03-30T12:05:08.850 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-03-30T12:05:08.853 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836490 2024-03-30T12:05:08.854 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-03-30T12:05:09.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:05:10.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:05:10.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:10.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-03-30T12:05:10.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:05:11.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T12:05:11.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:11.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836490 2024-03-30T12:05:11.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:11.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-03-30T12:05:11.723 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:11.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:05:11.724 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-03-30T12:05:11.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:11.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-03-30T12:05:11.726 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-03-30T12:05:11.726 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672964 2024-03-30T12:05:11.726 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:05:12.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672964 2024-03-30T12:05:12.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:05:12.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:12.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:12.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T12:05:12.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:05:12.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:05:12.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:05:12.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:05:12.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:05:12.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:05:12.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:05:12.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:05:12.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:05:12.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:12.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:12.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T12:05:12.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:05:12.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:05:12.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-03-30T12:05:12.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-30T12:05:13.369 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'foo' created 2024-03-30T12:05:13.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:05:14.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-03-30T12:05:14.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-30T12:05:14.851 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'csr_pool' created 2024-03-30T12:05:14.863 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:05:15.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-30T12:05:15.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:05:15.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:05:15.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:05:15.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:05:15.866 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:05:15.866 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:05:15.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:05:15.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:05:15.866 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:05:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:05:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:05:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:05:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:05:15.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:05:15.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:05:16.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:05:16.251 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:05:16.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:05:16.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:16.251 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:05:16.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-30T12:05:16.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-30T12:05:16.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-30T12:05:16.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:16.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:05:16.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-30T12:05:16.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-30T12:05:16.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672967' 2024-03-30T12:05:16.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:16.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-30T12:05:16.471 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:16.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:05:16.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-30T12:05:16.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:16.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-30T12:05:16.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-30T12:05:16.474 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836493 2024-03-30T12:05:16.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:16.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-30T12:05:16.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:16.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-30T12:05:16.759 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:16.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:05:16.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-30T12:05:16.761 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:16.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-30T12:05:16.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-30T12:05:16.762 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672967 2024-03-30T12:05:16.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:05:17.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-30T12:05:17.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:05:17.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:17.045 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:17.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:05:17.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:05:17.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:05:17.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:05:17.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:05:17.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:05:17.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:05:17.404 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:05:17.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:05:17.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:05:17.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:17.686 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:18.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:05:18.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:05:18.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:05:18.039 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-30T12:05:18.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-30T12:05:18.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-30T12:05:18.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-30T12:05:18.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:05:18.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T12:05:18.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-30T12:05:18.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T12:05:18.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T12:05:18.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T12:05:18.428 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T12:05:18.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T12:05:18.973 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:05:18.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:05:18.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:05:18.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:05:19.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-30T12:05:19.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-30T12:05:19.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-30T12:05:19.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-30T12:05:19.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-30T12:05:19.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:05:19.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-30T12:05:19.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-30T12:05:19.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T12:05:19.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T12:05:19.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T12:05:19.826 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T12:05:19.837 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T12:05:20.345 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:05:20.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:05:20.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:05:20.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:05:20.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-30T12:05:20.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-30T12:05:20.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-03-30T12:05:20.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-30T12:05:20.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-30T12:05:20.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-30T12:05:20.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:05:21.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-03-30T12:05:21.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-03-30T12:05:21.130 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-30T12:05:21.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-30T12:05:21.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-30T12:05:21.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T12:05:21.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-03-30T12:05:21.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-03-30T12:05:21.609 INFO:tasks.workunit.client.0.smithi003.stdout:created pool csr_pool snap snap1 2024-03-30T12:05:21.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-03-30T12:05:21.611 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-30T12:05:21.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-30T12:05:21.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-30T12:05:21.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-03-30T12:05:21.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-03-30T12:05:21.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-30T12:05:21.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-03-30T12:05:21.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:05:21.830 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:05:21.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:05:21.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:05:21.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:05:21.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:05:22.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:05:22.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-03-30T12:05:24.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:05:24.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:05:24.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:05:24.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:05:24.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:05:24.285 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:24.293+0000 7f8403614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:24.287 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:24.295+0000 7f8403614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:24.288 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:24.296+0000 7f8403614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:24.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:05:24.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:05:24.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:05:24.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:05:24.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:05:24.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:24.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:05:24.553 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:05:24.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:24.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:24.801 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:24.809+0000 7f8403614640 -1 Falling back to public interface 2024-03-30T12:05:24.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:25.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:25.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:25.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:05:25.833 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:05:25.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:25.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:26.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:05:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:05:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:27.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:27.208 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:27.215+0000 7f8403614640 -1 osd.1 30 log_to_monitors true 2024-03-30T12:05:27.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:28.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:28.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:28.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:05:28.430 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:05:28.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:28.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/603796121,v1:127.0.0.1:6811/603796121] [v2:127.0.0.1:6812/603796121,v1:127.0.0.1:6813/603796121] exists,up 0a9cd392-6867-4a93-bd1b-2bbb0199aada 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:05:28.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:05:28.948 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:05:28.948 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:05:28.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:05:28.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:05:28.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:05:29.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:05:29.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:05:29.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:05:29.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:05:29.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:05:29.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:05:29.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:05:29.344 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:05:29.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:05:29.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:29.345 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:05:29.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-03-30T12:05:29.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-03-30T12:05:29.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-03-30T12:05:29.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:29.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:05:29.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-03-30T12:05:29.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-03-30T12:05:29.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-146028888066' 2024-03-30T12:05:29.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:29.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-03-30T12:05:29.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:29.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:05:29.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-03-30T12:05:29.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:29.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-03-30T12:05:29.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-03-30T12:05:29.558 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 21474836496 2024-03-30T12:05:29.558 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:29.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-03-30T12:05:29.842 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:05:30.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:05:30.843 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836496 2024-03-30T12:05:31.136 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:31.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888066 2024-03-30T12:05:31.137 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:31.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:05:31.139 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888066 2024-03-30T12:05:31.139 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:31.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-03-30T12:05:31.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-03-30T12:05:31.140 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 146028888066 2024-03-30T12:05:31.140 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:05:31.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-03-30T12:05:31.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:05:31.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:31.423 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:31.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:05:31.799 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:05:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:05:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:05:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:05:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:05:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:05:31.800 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:05:32.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:05:32.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:05:32.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:32.079 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:32.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:05:32.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:05:32.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:05:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-30T12:05:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-30T12:05:32.433 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-03-30T12:05:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-03-30T12:05:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-30T12:05:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:05:32.577 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:05:32.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:05:32.578 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:05:32.578 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:05:32.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:05:32.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:05:32.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:05:32.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-03-30T12:05:34.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-30T12:05:34.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:05:34.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:05:34.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:05:34.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:05:34.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:05:34.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:05:34.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:05:34.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:05:34.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:05:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:05:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:05:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:05:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:05:34.752 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:05:34.752 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:05:34.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:05:34.753 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:05:34.753 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:05:34.754 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:34.754 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:05:34.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:05:34.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:05:34.773 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:34.781+0000 7fef3ae37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:34.775 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:34.783+0000 7fef3ae37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:34.777 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:34.784+0000 7fef3ae37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:35.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:05:35.040 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:05:35.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:35.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:05:35.295 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:35.303+0000 7fef3ae37640 -1 Falling back to public interface 2024-03-30T12:05:35.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:36.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:36.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:36.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:05:36.320 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:05:36.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:36.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:05:36.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:37.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:37.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:37.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:05:37.617 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:05:37.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:37.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:05:37.731 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:37.738+0000 7fef3ae37640 -1 osd.0 36 log_to_monitors true 2024-03-30T12:05:37.925 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:38.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:38.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:38.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:05:38.927 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:05:38.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:38.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/2684424431,v1:127.0.0.1:6803/2684424431] [v2:127.0.0.1:6804/2684424431,v1:127.0.0.1:6805/2684424431] exists,up 0481bb9e-33cb-4df0-b4fd-e4765ed9a728 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:05:39.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:05:39.253 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:05:39.253 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:05:39.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:05:39.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:05:39.253 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:05:39.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:05:39.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:05:39.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:05:39.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:05:39.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:05:39.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:05:39.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:05:39.624 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:05:39.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:05:39.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:39.624 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:05:39.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-30T12:05:39.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-30T12:05:39.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-03-30T12:05:39.729 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:05:39.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:05:39.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-03-30T12:05:39.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-03-30T12:05:39.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-146028888069' 2024-03-30T12:05:39.835 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:39.835 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-03-30T12:05:39.836 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:39.836 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:05:39.837 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-03-30T12:05:39.837 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:39.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-30T12:05:39.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-03-30T12:05:39.838 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 171798691842 2024-03-30T12:05:39.839 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:40.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-03-30T12:05:40.120 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:05:41.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:05:41.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:05:41.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-30T12:05:41.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:05:41.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-146028888069 2024-03-30T12:05:41.415 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:05:41.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:05:41.417 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-146028888069 2024-03-30T12:05:41.417 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:05:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-03-30T12:05:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2024-03-30T12:05:41.418 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 146028888069 2024-03-30T12:05:41.419 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:05:41.727 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-03-30T12:05:41.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:05:41.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:41.728 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:42.080 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-30T12:05:42.081 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:05:42.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:05:42.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:05:42.081 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:05:42.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:05:42.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:05:42.082 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:05:42.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-30T12:05:42.363 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:05:42.363 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:05:42.363 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:05:42.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-30T12:05:42.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:05:42.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:05:42.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-03-30T12:05:42.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-03-30T12:05:42.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-30T12:05:42.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-30T12:05:42.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:05:42.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:05:42.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:05:42.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:05:43.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:43.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-30T12:05:43.287 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:05:43.300 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:05:43.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:05:14.742138+0000 '>' 2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:43.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:05:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:05:44.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:05:44.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:05:44.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:05:44.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:05:44.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:05:44.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:05:44.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:05:14.742138+0000 '>' 2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:44.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:05:45.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:05:45.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:05:45.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:05:45.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:05:45.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:05:45.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:05:45.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:05:46.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:05:14.742138+0000 '>' 2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:46.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:05:47.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:05:47.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:05:47.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:05:47.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:05:47.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:05:47.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:05:47.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:05:47.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:05:14.742138+0000 '>' 2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:47.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:05:48.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:05:48.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:05:48.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-30T12:05:48.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-30T12:05:48.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:05:48.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:05:48.453 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-30T12:05:48.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:05:44.230140+0000 '>' 2024-03-30T12:05:14.742138+0000 2024-03-30T12:05:48.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:05:48.742 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-03-30T12:05:48.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-03-30T12:05:48.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-03-30T12:05:48.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-30T12:05:48.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-03-30T12:05:48.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T12:05:48.822 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-30T12:05:48.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: . as $in 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: | if type == "object" then 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: reduce keys[] as $key 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: else f 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr: end; 2024-03-30T12:05:48.845 INFO:tasks.workunit.client.0.smithi003.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-30T12:05:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-30T12:05:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-30T12:05:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-30T12:05:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-30T12:05:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-30T12:05:48.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-30T12:05:48.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T12:05:48.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-30T12:05:48.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-03-30T12:05:48.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-03-30T12:05:48.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-03-30T12:05:48.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-03-30T12:05:48.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-03-30T12:05:48.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-03-30T12:05:48.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-03-30T12:05:48.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-30T12:05:48.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-03-30T12:05:48.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-03-30T12:05:48.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-30T12:05:48.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-30T12:05:48.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-30T12:05:48.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-30T12:05:48.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-30T12:05:48.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-03-30T12:05:48.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-30T12:05:48.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-30T12:05:48.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-30T12:05:48.878 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-30T12:05:48.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-30T12:05:48.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-03-30T12:05:48.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-03-30T12:05:48.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-30T12:05:49.257 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'csr_pool' removed 2024-03-30T12:05:49.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T12:05:49.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:05:49.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:05:49.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:05:49.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:05:49.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:05:49.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:05:49.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:05:49.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:05:49.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:05:49.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:05:49.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:05:49.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:05:49.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:05:49.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:05:49.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:05:49.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:05:49.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:05:49.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:49.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:49.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:05:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:05:49.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:05:49.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:05:49.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:05:49.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:05:49.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:05:49.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:05:49.399 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:05:49.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:05:49.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:05:49.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:05:49.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:05:49.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:05:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:05:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:05:49.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:05:49.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:05:49.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:05:49.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:05:49.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:05:49.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:05:49.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:49.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:49.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:05:49.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:05:49.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:05:49.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T12:05:49.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T12:05:49.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:49.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:49.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T12:05:49.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T12:05:49.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T12:05:49.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T12:05:49.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:49.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T12:05:49.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T12:05:49.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T12:05:49.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:05:49.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:05:49.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T12:05:49.983 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:49.984 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:05:49.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:05:49.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T12:05:50.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T12:05:50.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:05:50.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:05:50.068 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:05:50.069 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:05:50.070 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T12:05:50.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T12:05:50.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T12:05:50.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:05:50.959 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:05:50.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:05:50.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:50.960 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:50.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:50.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:05:50.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:05:50.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-30T12:05:50.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:05:50.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-30T12:05:50.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:05:50.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:05:50.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:05:50.993 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:05:50.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:05:50.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:05:50.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:05:50.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0aa6a7eb-090f-4d5f-b01f-0b015330e54c 2024-03-30T12:05:50.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0aa6a7eb-090f-4d5f-b01f-0b015330e54c' 2024-03-30T12:05:50.995 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 0aa6a7eb-090f-4d5f-b01f-0b015330e54c 2024-03-30T12:05:50.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:05:51.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfAAhmNOYcARAAhSLw+PQ1Uh7Lvya2SjZbvg== 2024-03-30T12:05:51.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfAAhmNOYcARAAhSLw+PQ1Uh7Lvya2SjZbvg=="}' 2024-03-30T12:05:51.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0aa6a7eb-090f-4d5f-b01f-0b015330e54c -i td/osd-scrub-repair/0/new.json 2024-03-30T12:05:51.214 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:05:51.224 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T12:05:51.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfAAhmNOYcARAAhSLw+PQ1Uh7Lvya2SjZbvg== --osd-uuid 0aa6a7eb-090f-4d5f-b01f-0b015330e54c 2024-03-30T12:05:51.253 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.261+0000 7f1d92ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:51.265 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.273+0000 7f1d92ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:51.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.274+0000 7f1d92ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:51.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.275+0000 7f1d92ac8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:05:51.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.275+0000 7f1d92ac8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:05:51.267 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.275+0000 7f1d92ac8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:05:51.357 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:51.365+0000 7f1d92ac8640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T12:05:56.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T12:05:56.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:05:56.958 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T12:05:56.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T12:05:56.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:05:57.342 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:05:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T12:05:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:05:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:05:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:05:57.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:05:57.368 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:57.376+0000 7fdef11a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:57.370 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:57.378+0000 7fdef11a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:57.371 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:57.379+0000 7fdef11a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:05:57.624 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:05:57.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:57.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:05:57.891 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:05:57.899+0000 7fdef11a8640 -1 Falling back to public interface 2024-03-30T12:05:57.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:05:58.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:05:58.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:05:58.906 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:05:58.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:05:58.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:05:58.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:05:59.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:00.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:00.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:00.206 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:06:00.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:06:00.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:00.207 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:00.222 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:00.229+0000 7fdef11a8640 -1 osd.0 0 log_to_monitors true 2024-03-30T12:06:00.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:01.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:01.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:01.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:06:01.667 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:06:01.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:01.667 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:02.050 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:03.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:03.051 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:06:03.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:03.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:06:03.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:03.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:03.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:06:04.346 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:06:04.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:04.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:04.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:05.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:05.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:05.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:06:05.644 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:06:05.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:05.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:05.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:06.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:06.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:06.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:06:06.986 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:06:06.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:06.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:07.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:08.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:08.281 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:08.282 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:06:08.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:06:08.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:08.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:08.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:09.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:09.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:09.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:06:09.581 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:06:09.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:09.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:09.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:10.988 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:10.989 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:06:10.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:10.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:06:10.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:10.990 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:11.328 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:12.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:12.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:12.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:06:12.330 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:06:12.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:12.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:12.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:13.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:13.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:13.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:06:13.953 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:06:13.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:13.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:14.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:15.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:15.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:15.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:06:15.370 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:06:15.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:15.371 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:16.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:17.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:17.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:17.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:06:17.093 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:06:17.093 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:17.094 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:17.502 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:18.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:18.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:18.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:06:18.504 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:06:18.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:18.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:19.187 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:20.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:20.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:20.189 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:06:20.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:06:20.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:20.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:20.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:21.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:21.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:21.917 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:06:21.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:06:21.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:21.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:06:22.211 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1431847515,v1:127.0.0.1:6803/1431847515] [v2:127.0.0.1:6804/1431847515,v1:127.0.0.1:6805/1431847515] exists,up 0aa6a7eb-090f-4d5f-b01f-0b015330e54c 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:06:22.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:06:22.217 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:06:22.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:06:22.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:06:22.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fef3de74-527a-4e8f-b73d-dba0287618e8 2024-03-30T12:06:22.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fef3de74-527a-4e8f-b73d-dba0287618e8' 2024-03-30T12:06:22.219 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 fef3de74-527a-4e8f-b73d-dba0287618e8 2024-03-30T12:06:22.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:06:22.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+AAhmQrNgDhAA79eu8e6lu6+w0Q87XY/Ufw== 2024-03-30T12:06:22.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+AAhmQrNgDhAA79eu8e6lu6+w0Q87XY/Ufw=="}' 2024-03-30T12:06:22.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fef3de74-527a-4e8f-b73d-dba0287618e8 -i td/osd-scrub-repair/1/new.json 2024-03-30T12:06:22.576 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:06:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T12:06:22.588 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+AAhmQrNgDhAA79eu8e6lu6+w0Q87XY/Ufw== --osd-uuid fef3de74-527a-4e8f-b73d-dba0287618e8 2024-03-30T12:06:22.616 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.624+0000 7f64f8b9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:22.660 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.668+0000 7f64f8b9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:22.661 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.669+0000 7f64f8b9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:22.662 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.670+0000 7f64f8b9e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:06:22.662 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.670+0000 7f64f8b9e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:06:22.662 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.670+0000 7f64f8b9e640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:06:22.735 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:22.743+0000 7f64f8b9e640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T12:06:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T12:06:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:06:28.768 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T12:06:28.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T12:06:28.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:06:29.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T12:06:29.139 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:06:29.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:06:29.140 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:06:29.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:06:29.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:06:29.164 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:29.172+0000 7f5551506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:29.175 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:29.182+0000 7f5551506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:29.176 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:29.184+0000 7f5551506640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:29.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T12:06:29.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:06:29.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:06:29.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:06:29.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:06:29.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:29.431 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:06:29.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:06:29.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:29.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:29.706 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:29.714+0000 7f5551506640 -1 Falling back to public interface 2024-03-30T12:06:29.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:30.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:30.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:30.711 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:06:30.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:06:30.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:30.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:31.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:32.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:32.004 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:32.005 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:06:32.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:06:32.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:32.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:32.036 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:32.044+0000 7f5551506640 -1 osd.1 0 log_to_monitors true 2024-03-30T12:06:32.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:33.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:33.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:33.302 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:06:33.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:06:33.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:33.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:33.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:34.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:34.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:34.596 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:06:34.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:06:34.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:34.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:34.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:35.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:35.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:35.939 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:06:35.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:06:35.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:35.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:36.296 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:37.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:37.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:37.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:06:37.298 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:06:37.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:37.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:37.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:38.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:38.594 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:38.594 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:06:38.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:06:38.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:38.595 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:38.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:39.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:39.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:39.949 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:06:39.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:06:39.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:39.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:40.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:41.330 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:41.331 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:06:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:06:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:41.331 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:41.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:42.622 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:42.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:42.623 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:06:42.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:06:42.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:42.623 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:42.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:43.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:43.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:43.950 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:06:43.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:06:43.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:43.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:44.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:45.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:45.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:45.353 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:06:45.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:06:45.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:45.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:46.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:47.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:47.104 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:47.105 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:06:47.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:06:47.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:47.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:47.514 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:48.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:48.515 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:48.515 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:06:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:06:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:48.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:49.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:50.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:50.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:50.235 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:06:50.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:06:50.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:50.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:50.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:51.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:51.937 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:06:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:06:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:51.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:52.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:53.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:53.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:53.411 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:06:53.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:06:53.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:53.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:54.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:06:55.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:06:55.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:06:55.068 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:06:55.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:06:55.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:06:55.069 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:06:55.389 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3564700482,v1:127.0.0.1:6811/3564700482] [v2:127.0.0.1:6812/3564700482,v1:127.0.0.1:6813/3564700482] exists,up fef3de74-527a-4e8f-b73d-dba0287618e8 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T12:06:55.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:06:55.395 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:06:55.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T12:06:55.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:06:55.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33af2a14-ca7e-444c-8be7-b6f91c4c8bb8 2024-03-30T12:06:55.397 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 33af2a14-ca7e-444c-8be7-b6f91c4c8bb8 2024-03-30T12:06:55.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 33af2a14-ca7e-444c-8be7-b6f91c4c8bb8' 2024-03-30T12:06:55.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:06:55.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfAAhmzP32GBAATyR0a/jJfv/bIsyfb5DB/g== 2024-03-30T12:06:55.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfAAhmzP32GBAATyR0a/jJfv/bIsyfb5DB/g=="}' 2024-03-30T12:06:55.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33af2a14-ca7e-444c-8be7-b6f91c4c8bb8 -i td/osd-scrub-repair/2/new.json 2024-03-30T12:06:55.784 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:06:55.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T12:06:55.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfAAhmzP32GBAATyR0a/jJfv/bIsyfb5DB/g== --osd-uuid 33af2a14-ca7e-444c-8be7-b6f91c4c8bb8 2024-03-30T12:06:55.823 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.831+0000 7f670f211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:55.885 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.893+0000 7f670f211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:55.887 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.894+0000 7f670f211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:06:55.887 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.894+0000 7f670f211640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:06:55.887 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.894+0000 7f670f211640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:06:55.887 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.894+0000 7f670f211640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:06:55.985 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:06:55.993+0000 7f670f211640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T12:07:02.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T12:07:02.261 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:07:02.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T12:07:02.262 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T12:07:02.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:07:02.627 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T12:07:02.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T12:07:02.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:07:02.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:07:02.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:07:02.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:07:02.653 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:02.661+0000 7f44c0813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:02.655 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:02.663+0000 7f44c0813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:02.656 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:02.664+0000 7f44c0813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:02.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:07:02.910 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:07:02.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:02.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:03.178 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:03.186+0000 7f44c0813640 -1 Falling back to public interface 2024-03-30T12:07:03.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:04.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:04.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:04.239 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:07:04.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:07:04.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:04.240 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:04.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:05.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:05.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:05.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:07:05.531 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:07:05.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:05.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:05.545 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:05.553+0000 7f44c0813640 -1 osd.2 0 log_to_monitors true 2024-03-30T12:07:05.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:06.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:06.862 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:07:06.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:06.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:07:06.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:06.862 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:07.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:08.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:08.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:08.159 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:07:08.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:07:08.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:08.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:08.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:09.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:09.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:09.531 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:07:09.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:07:09.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:09.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:09.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:10.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:10.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:10.827 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:07:10.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:07:10.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:10.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:11.128 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:12.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:12.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:12.130 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:07:12.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:07:12.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:12.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:12.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:13.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:13.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:13.533 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:07:13.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:07:13.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:13.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:13.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:14.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:14.829 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:14.829 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:07:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:07:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:14.830 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:15.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:16.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:16.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:16.180 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:07:16.180 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:07:16.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:16.181 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:16.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:17.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:17.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:17.558 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:07:17.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:07:17.558 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:17.559 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:18.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:19.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:19.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:19.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:07:19.267 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:07:19.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:19.268 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:19.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:20.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:20.960 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:07:20.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:20.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:07:20.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:20.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:21.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:22.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:22.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:22.312 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:07:22.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:07:22.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:22.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:22.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:23.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:23.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:23.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:07:23.892 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:07:23.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:23.893 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:24.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:25.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:25.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:25.419 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:07:25.419 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:07:25.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:25.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:26.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:27.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:27.175 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:07:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:07:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:27.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:28.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:28.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:28.473 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:07:28.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:07:28.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:28.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:07:28.766 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3148496322,v1:127.0.0.1:6819/3148496322] [v2:127.0.0.1:6820/3148496322,v1:127.0.0.1:6821/3148496322] exists,up 33af2a14-ca7e-444c-8be7-b6f91c4c8bb8 2024-03-30T12:07:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:07:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:07:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:07:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-30T12:07:28.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T12:07:29.041 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T12:07:29.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T12:07:29.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T12:07:29.533 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T12:07:29.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:07:30.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T12:07:32.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-30T12:07:32.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:07:32.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:07:32.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:07:32.756 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:07:32.757 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:07:32.757 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:07:32.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:07:32.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:07:32.757 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:07:32.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:07:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:07:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:07:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:07:32.858 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:07:32.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:07:33.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:07:33.143 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:07:33.143 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:07:33.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:07:33.143 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:07:33.143 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:07:33.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-30T12:07:33.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-30T12:07:33.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-30T12:07:33.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:07:33.249 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:07:33.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-30T12:07:33.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-30T12:07:33.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-03-30T12:07:33.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:07:33.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:07:33.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-30T12:07:33.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-30T12:07:33.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509444' 2024-03-30T12:07:33.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:07:33.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-30T12:07:33.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:07:33.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:07:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-30T12:07:33.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:07:33.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-30T12:07:33.465 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836501 2024-03-30T12:07:33.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-30T12:07:33.467 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:07:33.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-30T12:07:33.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:07:34.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:07:34.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:07:35.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-30T12:07:35.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:07:36.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T12:07:36.044 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:07:36.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-30T12:07:36.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:07:36.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-30T12:07:36.339 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:07:36.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:07:36.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-30T12:07:36.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:07:36.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-30T12:07:36.342 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672975 2024-03-30T12:07:36.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-30T12:07:36.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:07:36.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-30T12:07:36.624 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:07:36.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-30T12:07:36.625 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:07:36.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:07:36.626 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-30T12:07:36.627 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:07:36.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-30T12:07:36.628 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-30T12:07:36.628 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509444 2024-03-30T12:07:36.628 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:07:36.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-30T12:07:36.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:07:36.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:07:36.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:07:37.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T12:07:37.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:07:37.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:07:37.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:07:37.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:07:37.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:07:37.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:07:37.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:07:37.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:07:37.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:07:37.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:07:37.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:07:37.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T12:07:37.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:07:37.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:07:37.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-30T12:07:37.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T12:07:37.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T12:07:37.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-30T12:07:37.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T12:07:37.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-30T12:07:38.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T12:07:38.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T12:07:39.058 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T12:07:39.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:07:40.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-30T12:07:40.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T12:07:40.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:07:40.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:07:40.073 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:07:40.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:07:40.074 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:07:40.074 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:07:40.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:07:40.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:07:40.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:07:40.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:07:40.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:07:40.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:07:40.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:07:40.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:07:40.175 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:07:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:07:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:07:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:07:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:07:40.462 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:07:40.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:07:40.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-30T12:07:40.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-30T12:07:40.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-30T12:07:40.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:07:40.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:07:40.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-30T12:07:40.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-30T12:07:40.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978' 2024-03-30T12:07:40.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:07:40.749 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:07:40.879 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-30T12:07:40.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-30T12:07:40.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509446' 2024-03-30T12:07:40.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:07:40.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-30T12:07:40.880 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:07:40.880 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:07:40.881 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-30T12:07:40.882 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:07:40.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-30T12:07:40.883 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836503 2024-03-30T12:07:40.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-30T12:07:40.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:07:41.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-30T12:07:41.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:07:42.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:07:42.180 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:07:42.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-30T12:07:42.470 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:07:42.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-30T12:07:42.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:07:42.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:07:42.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-30T12:07:42.474 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:07:42.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-30T12:07:42.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-30T12:07:42.475 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672978 2024-03-30T12:07:42.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:07:42.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-30T12:07:42.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:07:42.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-30T12:07:42.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:07:42.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:07:42.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-30T12:07:42.795 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:07:42.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-30T12:07:42.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-30T12:07:42.797 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509446 2024-03-30T12:07:42.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:07:43.079 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-30T12:07:43.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:07:43.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:07:43.080 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:07:43.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T12:07:43.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:07:43.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:07:43.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:07:43.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:07:43.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:07:43.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:07:43.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:07:43.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T12:07:43.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:07:43.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:07:43.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:07:44.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T12:07:44.066 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:07:44.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:07:44.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T12:07:44.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-30T12:07:44.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:07:44.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T12:07:44.067 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-30T12:07:44.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T12:07:44.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T12:07:44.068 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T12:07:44.447 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T12:07:44.458 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T12:07:44.983 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:07:44.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:07:44.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:07:44.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:07:45.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-30T12:07:45.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T12:07:45.238 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-30T12:07:45.238 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-30T12:07:45.239 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:07:45.542 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T12:07:45.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T12:07:45.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T12:07:45.919 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T12:07:45.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T12:07:46.438 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:07:46.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:07:46.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:07:46.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:07:46.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-30T12:07:46.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T12:07:46.744 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-30T12:07:46.744 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-30T12:07:46.745 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-30T12:07:47.034 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-30T12:07:47.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:07:47.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:07:47.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:07:47.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:07:47.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:07:47.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:07:47.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:07:47.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:07:47.344 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:07:48.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:07:48.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:07:48.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T12:07:48.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:07:48.653 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:07:48.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:07:48.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:07:48.654 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:07:48.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:07:48.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-30T12:07:48.962 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-30T12:07:50.040 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-30T12:07:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-30T12:07:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-30T12:07:51.686 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-30T12:07:52.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-30T12:07:52.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-30T12:07:52.955 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-30T12:07:53.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-30T12:07:53.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-30T12:07:54.668 INFO:tasks.workunit.client.0.smithi003.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-30T12:07:55.183 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:07:55.188 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:07:55.189 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:07:55.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:07:55.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:07:55.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:07:55.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:07:55.191 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:07:55.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:07:55.191 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:07:55.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:07:55.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:07:55.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:07:55.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:07:55.217 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:55.225+0000 7f678b815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:55.219 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:55.227+0000 7f678b815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:55.221 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:55.228+0000 7f678b815640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:55.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:07:55.735 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:55.743+0000 7f678b815640 -1 Falling back to public interface 2024-03-30T12:07:55.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:56.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:56.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:56.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:07:56.766 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:07:56.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:56.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:07:57.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:58.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:58.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:58.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:07:58.057 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:07:58.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:58.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:07:58.132 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:58.140+0000 7f678b815640 -1 osd.0 35 log_to_monitors true 2024-03-30T12:07:58.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:07:59.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:07:59.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:59.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:07:59.379 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:07:59.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:59.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 42 up_thru 42 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/2447502461,v1:127.0.0.1:6803/2447502461] [v2:127.0.0.1:6804/2447502461,v1:127.0.0.1:6805/2447502461] exists,up 0aa6a7eb-090f-4d5f-b01f-0b015330e54c 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:07:59.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:07:59.674 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:07:59.675 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:07:59.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:07:59.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:07:59.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:07:59.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:07:59.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:07:59.699 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:59.706+0000 7f3751d93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:59.700 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:59.708+0000 7f3751d93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:59.701 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:07:59.709+0000 7f3751d93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:07:59.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:07:59.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:07:59.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:07:59.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:07:59.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:07:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:07:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:07:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:07:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:07:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:08:00.216 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:00.223+0000 7f3751d93640 -1 Falling back to public interface 2024-03-30T12:08:00.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:08:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:08:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:01.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:08:01.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:08:02.543 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:08:02.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:02.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:08:02.561 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:02.568+0000 7f3751d93640 -1 osd.1 36 log_to_monitors true 2024-03-30T12:08:02.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:03.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:03.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:03.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:08:03.873 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:08:03.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:03.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/4116108096,v1:127.0.0.1:6811/4116108096] [v2:127.0.0.1:6812/4116108096,v1:127.0.0.1:6813/4116108096] exists,up fef3de74-527a-4e8f-b73d-dba0287618e8 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:08:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:08:04.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T12:08:04.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T12:08:04.221 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.2 2024-03-30T12:08:04.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:08:04.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T12:08:04.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T12:08:04.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:08:04.221 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:08:04.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:08:04.243 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:04.251+0000 7f3cbbe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:04.259 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:04.267+0000 7f3cbbe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:04.260 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:04.268+0000 7f3cbbe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:04.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:04.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:08:04.774 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:04.782+0000 7f3cbbe0f640 -1 Falling back to public interface 2024-03-30T12:08:04.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:05.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:05.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:05.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:08:05.843 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:08:05.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:05.844 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:08:06.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:07.134 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:07.142+0000 7f3cbbe0f640 -1 osd.2 39 log_to_monitors true 2024-03-30T12:08:07.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:07.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:07.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:07.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:08:07.537 INFO:tasks.workunit.client.0.smithi003.stderr:osd.2 up in weight 1 up_from 49 up_thru 37 down_at 40 last_clean_interval [15,39) [v2:127.0.0.1:6818/1746020280,v1:127.0.0.1:6819/1746020280] [v2:127.0.0.1:6820/1746020280,v1:127.0.0.1:6821/1746020280] exists,up 33af2a14-ca7e-444c-8be7-b6f91c4c8bb8 2024-03-30T12:08:07.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:08:07.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:08:07.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:08:07.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-30T12:08:07.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T12:08:08.606 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' removed 2024-03-30T12:08:08.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T12:08:08.621 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T12:08:09.152 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T12:08:09.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:08:10.165 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T12:08:11.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-30T12:08:11.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:08:11.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:08:11.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:08:11.618 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:08:11.618 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:08:11.619 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:08:11.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:08:11.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:08:11.619 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:08:11.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:08:11.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:08:11.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:08:11.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:08:11.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:08:11.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:08:11.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:08:11.957 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:08:11.957 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:08:11.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:08:11.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:08:11.958 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:08:12.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626436 2024-03-30T12:08:12.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626436 2024-03-30T12:08:12.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-180388626436' 2024-03-30T12:08:12.065 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:08:12.065 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:08:12.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528323 2024-03-30T12:08:12.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528323 2024-03-30T12:08:12.170 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-180388626436 1-193273528323' 2024-03-30T12:08:12.171 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:08:12.171 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:08:12.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-03-30T12:08:12.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-03-30T12:08:12.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-180388626436 1-193273528323 2-210453397506' 2024-03-30T12:08:12.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:08:12.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-180388626436 2024-03-30T12:08:12.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:08:12.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:08:12.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-180388626436 2024-03-30T12:08:12.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:08:12.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626436 2024-03-30T12:08:12.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 180388626436' 2024-03-30T12:08:12.279 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 180388626436 2024-03-30T12:08:12.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:08:12.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626435 -lt 180388626436 2024-03-30T12:08:12.562 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:08:13.563 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:08:13.563 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:08:13.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626436 -lt 180388626436 2024-03-30T12:08:13.857 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:08:13.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528323 2024-03-30T12:08:13.858 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:08:13.859 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:08:13.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528323 2024-03-30T12:08:13.859 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:08:13.860 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528323 2024-03-30T12:08:13.861 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528323' 2024-03-30T12:08:13.861 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 193273528323 2024-03-30T12:08:13.861 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:08:14.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528324 -lt 193273528323 2024-03-30T12:08:14.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:08:14.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-210453397506 2024-03-30T12:08:14.156 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:08:14.157 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:08:14.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-210453397506 2024-03-30T12:08:14.158 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:08:14.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-03-30T12:08:14.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 210453397506' 2024-03-30T12:08:14.159 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 210453397506 2024-03-30T12:08:14.159 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:08:14.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397507 -lt 210453397506 2024-03-30T12:08:14.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:08:14.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:08:14.441 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:08:14.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T12:08:14.793 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:08:14.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:08:14.793 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:08:14.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:08:14.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:08:14.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:08:14.794 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:08:15.071 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T12:08:15.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:08:15.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:08:15.072 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:08:15.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T12:08:15.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:08:15.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:08:15.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-30T12:08:15.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T12:08:15.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-30T12:08:15.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-30T12:08:15.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:08:15.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:08:15.725 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:08:16.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T12:07:38.890603+0000 2024-03-30T12:08:16.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T12:08:16.329 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T12:07:38.890603+0000 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:07:38.890603+0000 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:08:16.341 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:08:16.342 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:08:16.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:07:38.890603+0000 '>' 2024-03-30T12:07:38.890603+0000 2024-03-30T12:08:16.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:08:17.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:08:17.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:08:17.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:08:17.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:08:17.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:08:17.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:08:17.621 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:08:17.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:07:38.890603+0000 '>' 2024-03-30T12:07:38.890603+0000 2024-03-30T12:08:17.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:08:18.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:08:18.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:08:18.910 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:08:18.910 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:08:18.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:08:18.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:08:18.911 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:08:19.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:08:17.081347+0000 '>' 2024-03-30T12:07:38.890603+0000 2024-03-30T12:08:19.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:08:19.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-30T12:08:19.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-30T12:08:19.204 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-30T12:08:19.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-30T12:08:19.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-30T12:08:19.205 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-30T12:08:19.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-30T12:08:19.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-30T12:08:19.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-30T12:08:19.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:08:19.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:08:19.459 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:08:19.459 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:08:19.460 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:08:19.460 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:08:19.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:08:19.461 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:08:19.462 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:08:19.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:08:19.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:08:19.463 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:08:19.463 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:08:19.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:08:19.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:08:19.465 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:08:19.481 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:08:19.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:19.482 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:19.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:08:19.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:08:19.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:08:19.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:08:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:08:19.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:08:19.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:08:19.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:08:19.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:08:19.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:08:19.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:08:19.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:08:19.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:08:19.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:08:19.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:08:19.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:08:19.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:08:19.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:08:19.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T12:08:19.495 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T12:08:19.495 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:19.495 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:19.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T12:08:19.497 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T12:08:19.498 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T12:08:19.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T12:08:19.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T12:08:19.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:19.893 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:19.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T12:08:20.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T12:08:20.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T12:08:20.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:08:20.086 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:08:20.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T12:08:20.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T12:08:20.087 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:08:20.087 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:08:20.087 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:08:20.088 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:08:20.088 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:20.088 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:20.088 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:08:20.088 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:08:20.088 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T12:08:20.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T12:08:20.172 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:20.173 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:08:20.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:08:20.174 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T12:08:20.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-30T12:08:20.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T12:08:20.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T12:08:20.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T12:08:20.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T12:08:20.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T12:08:20.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:21.071 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:08:21.072 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:08:21.101 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-30T12:08:21.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:08:21.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-30T12:08:21.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:08:21.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:08:21.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T12:08:21.105 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:08:21.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:08:21.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:08:21.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:08:21.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:08:21.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:08:21.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:08:21.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:08:21.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:08:21.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:08:21.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e25260f-febe-4eba-90c9-fb065017ea69 2024-03-30T12:08:21.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6e25260f-febe-4eba-90c9-fb065017ea69' 2024-03-30T12:08:21.109 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 6e25260f-febe-4eba-90c9-fb065017ea69 2024-03-30T12:08:21.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:08:21.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1AAhmhfboBxAA8Fdrzhhru79qYGCDuimMbg== 2024-03-30T12:08:21.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1AAhmhfboBxAA8Fdrzhhru79qYGCDuimMbg=="}' 2024-03-30T12:08:21.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e25260f-febe-4eba-90c9-fb065017ea69 -i td/osd-scrub-repair/0/new.json 2024-03-30T12:08:21.350 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:08:21.364 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T12:08:21.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1AAhmhfboBxAA8Fdrzhhru79qYGCDuimMbg== --osd-uuid 6e25260f-febe-4eba-90c9-fb065017ea69 2024-03-30T12:08:21.395 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.402+0000 7f663a528640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:21.401 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.409+0000 7f663a528640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:21.402 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.410+0000 7f663a528640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.410+0000 7f663a528640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:08:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.410+0000 7f663a528640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:08:21.403 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.410+0000 7f663a528640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:08:21.493 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:21.501+0000 7f663a528640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T12:08:27.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T12:08:27.096 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:08:27.097 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T12:08:27.097 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T12:08:27.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:08:27.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T12:08:27.479 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:08:27.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:08:27.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:08:27.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:08:27.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:08:27.505 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:27.512+0000 7f1e0460f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:27.507 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:27.514+0000 7f1e0460f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:27.508 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:27.515+0000 7f1e0460f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:08:27.763 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:08:27.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:27.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:28.022 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:28.029+0000 7f1e0460f640 -1 Falling back to public interface 2024-03-30T12:08:28.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:29.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:29.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:29.044 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:08:29.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:08:29.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:29.045 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:29.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:30.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:30.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:30.340 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:08:30.340 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:08:30.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:30.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:30.359 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:30.365+0000 7f1e0460f640 -1 osd.0 0 log_to_monitors true 2024-03-30T12:08:30.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:31.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:31.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:31.646 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:08:31.646 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:08:31.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:31.647 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:31.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:32.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:32.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:32.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:08:32.941 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:08:32.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:32.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:33.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:34.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:34.262 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:08:34.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:08:34.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:34.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:35.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:35.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:35.555 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:08:35.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:08:35.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:35.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:35.894 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:36.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:36.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:36.896 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:08:36.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:08:36.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:36.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:37.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:38.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:38.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:38.272 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:08:38.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:08:38.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:38.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:38.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:39.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:39.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:39.572 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:08:39.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:08:39.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:39.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:39.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:40.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:40.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:40.921 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:08:40.921 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:08:40.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:40.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:41.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:42.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:42.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:42.276 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:08:42.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:08:42.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:42.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:42.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:43.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:43.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:43.999 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:08:44.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:08:44.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:44.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:44.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:45.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:45.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:45.449 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:08:45.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:08:45.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:45.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:46.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:47.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:47.234 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:08:47.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:47.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:08:47.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:47.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:48.295 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:49.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:49.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:49.297 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:08:49.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:08:49.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:49.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:50.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:51.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:51.016 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:08:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:08:51.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:51.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:51.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:08:52.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:08:52.574 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:08:52.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:08:52.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:08:52.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:08:52.575 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:08:52.870 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3848061441,v1:127.0.0.1:6803/3848061441] [v2:127.0.0.1:6804/3848061441,v1:127.0.0.1:6805/3848061441] exists,up 6e25260f-febe-4eba-90c9-fb065017ea69 2024-03-30T12:08:52.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:08:52.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:08:52.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7636ea4-148a-4237-aa1e-9fb22849970e 2024-03-30T12:08:52.876 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 a7636ea4-148a-4237-aa1e-9fb22849970e 2024-03-30T12:08:52.877 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a7636ea4-148a-4237-aa1e-9fb22849970e' 2024-03-30T12:08:52.877 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:08:52.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUAAhmGv2pNRAAd5zuWv/k/cuWgvDjEAUvJQ== 2024-03-30T12:08:52.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUAAhmGv2pNRAAd5zuWv/k/cuWgvDjEAUvJQ=="}' 2024-03-30T12:08:52.892 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7636ea4-148a-4237-aa1e-9fb22849970e -i td/osd-scrub-repair/1/new.json 2024-03-30T12:08:53.251 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:08:53.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T12:08:53.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUAAhmGv2pNRAAd5zuWv/k/cuWgvDjEAUvJQ== --osd-uuid a7636ea4-148a-4237-aa1e-9fb22849970e 2024-03-30T12:08:53.290 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.298+0000 7fdf42fe1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:53.335 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.343+0000 7fdf42fe1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:53.336 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.344+0000 7fdf42fe1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:53.336 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.344+0000 7fdf42fe1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:08:53.336 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.344+0000 7fdf42fe1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:08:53.337 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.344+0000 7fdf42fe1640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:08:53.410 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:08:53.418+0000 7fdf42fe1640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T12:08:59.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T12:08:59.602 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:08:59.603 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T12:08:59.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T12:08:59.603 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:08:59.968 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:08:59.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T12:08:59.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:08:59.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:08:59.969 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:08:59.970 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:08:59.994 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:00.002+0000 7f0d32fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:59.996 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:00.004+0000 7f0d32fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:08:59.997 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:00.005+0000 7f0d32fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:00.263 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:09:00.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:09:00.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:00.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:00.523 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:00.531+0000 7f0d32fff640 -1 Falling back to public interface 2024-03-30T12:09:00.544 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:01.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:01.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:01.546 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:09:01.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:09:01.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:01.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:01.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:02.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:02.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:02.840 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:09:02.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:09:02.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:02.841 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:02.851 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:02.859+0000 7f0d32fff640 -1 osd.1 0 log_to_monitors true 2024-03-30T12:09:03.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:04.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:04.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:04.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:09:04.160 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:09:04.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:04.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:04.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:05.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:05.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:05.454 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:09:05.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:09:05.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:05.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:05.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:06.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:06.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:06.902 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:09:06.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:09:06.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:06.902 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:07.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:08.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:08.195 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:09:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:09:08.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:08.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:08.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:09.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:09.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:09.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:09:09.486 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:09:09.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:09.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:09.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:10.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:10.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:10.866 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:09:10.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:09:10.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:10.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:11.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:12.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:12.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:12.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:09:12.163 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:09:12.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:12.163 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:12.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:13.457 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:13.564 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:13.565 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:09:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:09:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:13.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:13.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:14.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:14.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:14.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:09:14.750 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:09:14.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:14.750 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:15.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:16.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:16.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:16.042 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:09:16.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:09:16.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:16.043 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:16.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:17.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:17.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:17.358 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:09:17.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:09:17.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:17.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:17.790 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:18.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:18.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:18.792 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:09:18.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:09:18.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:18.792 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:19.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:20.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:20.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:20.442 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:09:20.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:09:20.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:20.443 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:21.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:22.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:22.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:22.199 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:09:22.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:09:22.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:22.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:22.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:23.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:23.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:23.493 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:09:23.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:09:23.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:23.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:24.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:25.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:25.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:25.168 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:09:25.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:09:25.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:25.169 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:25.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:26.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:26.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:26.466 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T12:09:26.466 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T12:09:26.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:26.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:09:26.757 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1844950321,v1:127.0.0.1:6811/1844950321] [v2:127.0.0.1:6812/1844950321,v1:127.0.0.1:6813/1844950321] exists,up a7636ea4-148a-4237-aa1e-9fb22849970e 2024-03-30T12:09:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:09:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:09:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:09:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:09:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-30T12:09:26.761 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T12:09:26.762 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:09:26.763 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T12:09:26.764 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:09:26.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bbb4ad05-744f-40a0-912d-0a80454cb35e 2024-03-30T12:09:26.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bbb4ad05-744f-40a0-912d-0a80454cb35e' 2024-03-30T12:09:26.765 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 bbb4ad05-744f-40a0-912d-0a80454cb35e 2024-03-30T12:09:26.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:09:26.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2AAhmfGjgLhAAlSoZDJb32I/rpNvs+feG5Q== 2024-03-30T12:09:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2AAhmfGjgLhAAlSoZDJb32I/rpNvs+feG5Q=="}' 2024-03-30T12:09:26.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bbb4ad05-744f-40a0-912d-0a80454cb35e -i td/osd-scrub-repair/2/new.json 2024-03-30T12:09:27.147 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:09:27.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T12:09:27.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2AAhmfGjgLhAAlSoZDJb32I/rpNvs+feG5Q== --osd-uuid bbb4ad05-744f-40a0-912d-0a80454cb35e 2024-03-30T12:09:27.188 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.196+0000 7f8f7f319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:27.232 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.240+0000 7f8f7f319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:27.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.241+0000 7f8f7f319640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:27.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.241+0000 7f8f7f319640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:09:27.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.241+0000 7f8f7f319640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:09:27.233 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.241+0000 7f8f7f319640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:09:27.307 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:27.315+0000 7f8f7f319640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T12:09:33.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T12:09:33.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:09:33.291 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T12:09:33.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T12:09:33.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:09:33.662 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T12:09:33.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T12:09:33.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:09:33.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:09:33.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:09:33.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:09:33.689 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:33.697+0000 7f1f57ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:33.691 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:33.699+0000 7f1f57ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:33.692 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:33.700+0000 7f1f57ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:33.947 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:09:33.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:09:33.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:33.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:34.218 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:34.226+0000 7f1f57ac8640 -1 Falling back to public interface 2024-03-30T12:09:34.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:35.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:35.236 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:35.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:09:35.237 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:09:35.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:35.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:35.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:36.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:36.532 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:09:36.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:09:36.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:36.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:36.711 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:09:36.717+0000 7f1f57ac8640 -1 osd.2 0 log_to_monitors true 2024-03-30T12:09:36.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:37.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:37.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:37.825 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:09:37.825 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:09:37.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:37.826 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:38.116 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:39.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:39.117 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:39.118 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:09:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:09:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:39.118 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:39.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:40.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:40.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:40.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:09:40.413 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:09:40.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:40.413 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:40.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:41.758 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:41.759 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:09:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:09:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:41.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:42.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:43.052 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:43.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:43.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:09:43.053 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:09:43.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:43.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:43.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:44.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:44.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:44.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:09:44.349 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:09:44.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:44.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:44.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:45.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:45.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:45.641 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:09:45.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:09:45.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:45.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:45.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:46.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:46.938 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:46.938 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:09:46.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:09:46.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:46.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:47.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:48.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:48.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:48.232 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:09:48.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:09:48.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:48.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:48.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:49.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:49.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:49.644 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:09:49.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:09:49.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:49.645 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:50.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:51.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:51.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:51.248 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:09:51.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:09:51.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:51.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:51.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:52.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:52.992 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:52.993 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:09:52.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:09:52.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:52.993 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:53.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:54.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:54.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:54.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:09:54.441 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:09:54.441 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:54.442 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:54.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:55.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:55.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:55.998 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:09:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:09:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:55.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:56.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:57.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:57.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:57.747 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:09:57.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:09:57.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:57.748 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:58.336 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:09:59.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:09:59.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:09:59.338 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:09:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:09:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:09:59.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:09:59.628 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3992971428,v1:127.0.0.1:6819/3992971428] [v2:127.0.0.1:6820/3992971428,v1:127.0.0.1:6821/3992971428] exists,up bbb4ad05-744f-40a0-912d-0a80454cb35e 2024-03-30T12:09:59.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:09:59.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:09:59.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:09:59.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-30T12:09:59.629 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T12:09:59.945 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T12:09:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T12:09:59.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T12:10:00.296 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T12:10:00.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:10:01.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T12:10:03.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-30T12:10:03.730 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:10:03.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:10:03.731 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:10:03.731 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:10:03.731 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:10:03.732 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:10:03.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:10:03.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:10:03.732 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:10:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:10:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:10:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:10:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:10:03.832 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:10:03.832 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:10:04.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:10:04.113 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:04.113 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:10:04.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:10:04.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:04.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:10:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-30T12:10:04.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-30T12:10:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-30T12:10:04.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:04.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:10:04.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-30T12:10:04.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-30T12:10:04.323 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672971' 2024-03-30T12:10:04.324 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:04.324 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:10:04.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-30T12:10:04.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-30T12:10:04.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672971 2-64424509444' 2024-03-30T12:10:04.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:04.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-30T12:10:04.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:04.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:10:04.430 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-30T12:10:04.431 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-30T12:10:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-30T12:10:04.432 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836503 2024-03-30T12:10:04.432 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:10:04.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-30T12:10:04.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:10:05.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:10:05.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:10:06.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-30T12:10:06.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:06.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-30T12:10:06.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:06.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:10:06.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-30T12:10:06.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:06.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-30T12:10:06.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-30T12:10:06.011 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672971 2024-03-30T12:10:06.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:10:06.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-30T12:10:06.297 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:06.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-30T12:10:06.298 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:06.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:10:06.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-30T12:10:06.299 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:06.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-30T12:10:06.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-30T12:10:06.301 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509444 2024-03-30T12:10:06.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:10:06.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-30T12:10:06.587 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:10:06.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:10:06.588 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:10:06.968 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:10:07.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:10:07.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:10:07.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:10:07.248 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:10:07.596 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T12:10:07.597 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-30T12:10:08.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-30T12:10:08.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-30T12:10:08.665 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'ecpool' created 2024-03-30T12:10:08.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:10:09.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-30T12:10:09.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-30T12:10:10.155 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 2 allow_ec_overwrites to true 2024-03-30T12:10:10.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T12:10:10.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:10:10.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:10:10.167 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:10:10.167 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:10:10.168 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:10:10.168 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:10:10.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:10:10.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:10:10.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:10:10.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:10:10.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:10:10.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:10:10.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:10:10.258 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:10:10.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:10:10.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:10:10.555 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:10.555 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:10:10.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:10:10.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:10.556 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:10:10.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-30T12:10:10.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-30T12:10:10.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-30T12:10:10.687 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:10.687 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:10:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-30T12:10:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-30T12:10:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672973' 2024-03-30T12:10:10.791 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:10.791 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:10:10.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-30T12:10:10.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-30T12:10:10.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672973 2-64424509446' 2024-03-30T12:10:10.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:10.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-30T12:10:10.897 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:10.898 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:10:10.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-30T12:10:10.899 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:10.899 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-30T12:10:10.899 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836505 2024-03-30T12:10:10.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-30T12:10:10.900 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:10:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-30T12:10:11.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:10:12.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:10:12.186 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:10:12.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-30T12:10:12.480 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:12.481 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-30T12:10:12.481 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:12.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:10:12.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-30T12:10:12.483 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:12.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-30T12:10:12.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-30T12:10:12.484 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672973 2024-03-30T12:10:12.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:10:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-30T12:10:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:12.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-30T12:10:12.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:12.766 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:10:12.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-30T12:10:12.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:12.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-30T12:10:12.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-30T12:10:12.768 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509446 2024-03-30T12:10:12.768 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:10:13.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-30T12:10:13.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:10:13.049 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:10:13.050 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:10:13.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:10:13.688 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T12:10:13.688 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:10:13.688 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:10:13.688 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:10:14.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T12:10:14.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:10:14.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:10:14.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T12:10:14.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T12:10:14.522 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T12:10:14.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T12:10:15.091 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:10:15.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:10:15.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:10:15.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:10:15.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-30T12:10:15.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T12:10:15.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-30T12:10:15.473 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-30T12:10:15.473 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-30T12:10:15.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-30T12:10:15.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:10:15.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-30T12:10:15.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-30T12:10:15.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-30T12:10:15.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-30T12:10:15.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-30T12:10:16.146 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is set 2024-03-30T12:10:16.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-30T12:10:16.689 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:10:16.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:10:16.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:10:16.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:10:17.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-30T12:10:17.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-30T12:10:17.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-30T12:10:17.013 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-30T12:10:17.013 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr:0' 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-30T12:10:17.305 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-30T12:10:17.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:10:17.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:10:17.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:10:17.307 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:10:17.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:10:17.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:10:17.307 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:10:17.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:10:17.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:10:17.614 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:10:17.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:10:17.615 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:10:17.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:10:17.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:10:17.615 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:10:18.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:10:18.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:10:18.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-30T12:10:18.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:10:18.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:10:18.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:10:18.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:10:18.924 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:10:19.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:10:19.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-30T12:10:19.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-30T12:10:20.424 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-30T12:10:20.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-30T12:10:20.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-30T12:10:22.079 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-30T12:10:22.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-30T12:10:22.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-30T12:10:23.389 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-30T12:10:23.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-30T12:10:23.904 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-30T12:10:25.060 INFO:tasks.workunit.client.0.smithi003.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-30T12:10:25.575 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:10:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:10:25.581 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:10:25.582 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:10:25.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:10:25.583 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:10:25.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:10:25.583 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:10:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:10:25.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:10:25.609 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:25.617+0000 7f25e2611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:25.611 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:25.619+0000 7f25e2611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:25.613 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:25.621+0000 7f25e2611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:10:25.875 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:10:25.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:25.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:10:26.137 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:26.145+0000 7f25e2611640 -1 Falling back to public interface 2024-03-30T12:10:26.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:10:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:27.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:10:27.467 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:28.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:28.468 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:10:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:10:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:28.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:10:28.576 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:28.583+0000 7f25e2611640 -1 osd.0 35 log_to_monitors true 2024-03-30T12:10:28.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:10:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:10:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:29.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/4198771305,v1:127.0.0.1:6803/4198771305] [v2:127.0.0.1:6804/4198771305,v1:127.0.0.1:6805/4198771305] exists,up 6e25260f-febe-4eba-90c9-fb065017ea69 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:10:30.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:10:30.061 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:10:30.062 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.1 2024-03-30T12:10:30.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:10:30.063 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:10:30.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:10:30.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:10:30.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:10:30.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:10:30.085 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:30.093+0000 7f42fb3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:30.088 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:30.095+0000 7f42fb3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:30.088 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:30.096+0000 7f42fb3bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:10:30.360 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:10:30.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:30.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:10:30.606 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:30.614+0000 7f42fb3bb640 -1 Falling back to public interface 2024-03-30T12:10:30.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:31.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:31.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:31.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:10:31.641 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:31.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:31.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:10:31.933 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:32.911 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:32.919+0000 7f42fb3bb640 -1 osd.1 36 log_to_monitors true 2024-03-30T12:10:32.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:32.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:32.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:10:32.935 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:10:32.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:32.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:10:33.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:34.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:34.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:34.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:10:34.244 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:10:34.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:34.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:10:34.551 INFO:tasks.workunit.client.0.smithi003.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1933072005,v1:127.0.0.1:6811/1933072005] [v2:127.0.0.1:6812/1933072005,v1:127.0.0.1:6813/1933072005] exists,up a7636ea4-148a-4237-aa1e-9fb22849970e 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:10:34.552 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:34.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-30T12:10:34.554 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.2 2024-03-30T12:10:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-30T12:10:34.555 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-30T12:10:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-30T12:10:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:10:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:10:34.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:10:34.576 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:34.584+0000 7fa96b738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:34.602 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:34.610+0000 7fa96b738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:34.603 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:34.611+0000 7fa96b738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:10:34.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:34.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:10:35.117 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:35.125+0000 7fa96b738640 -1 Falling back to public interface 2024-03-30T12:10:35.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:36.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:36.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:36.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:10:36.158 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:36.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:36.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:10:36.454 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:37.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:37.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:37.455 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:10:37.456 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:10:37.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:37.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:10:37.551 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:37.559+0000 7fa96b738640 -1 osd.2 38 log_to_monitors true 2024-03-30T12:10:37.749 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:38.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:38.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:38.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:10:38.751 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:10:38.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:38.751 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:10:39.046 INFO:tasks.workunit.client.0.smithi003.stderr:osd.2 up in weight 1 up_from 48 up_thru 48 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/695944455,v1:127.0.0.1:6819/695944455] [v2:127.0.0.1:6820/695944455,v1:127.0.0.1:6821/695944455] exists,up bbb4ad05-744f-40a0-912d-0a80454cb35e 2024-03-30T12:10:39.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:10:39.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:10:39.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:10:39.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-30T12:10:39.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T12:10:39.561 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' removed 2024-03-30T12:10:39.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T12:10:39.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T12:10:39.946 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T12:10:39.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:10:40.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T12:10:42.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-30T12:10:42.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:10:42.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:10:42.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:10:42.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:10:42.450 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:10:42.450 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:10:42.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:10:42.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:10:42.450 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:10:42.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:10:42.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:10:42.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:10:42.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:10:42.551 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:10:42.551 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:10:42.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:10:42.845 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:10:42.845 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:10:42.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:10:42.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:42.846 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:10:42.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659140 2024-03-30T12:10:42.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659140 2024-03-30T12:10:42.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140' 2024-03-30T12:10:42.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:42.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:10:43.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-03-30T12:10:43.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-03-30T12:10:43.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-03-30T12:10:43.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:10:43.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:10:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430211 2024-03-30T12:10:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430211 2024-03-30T12:10:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-206158430211' 2024-03-30T12:10:43.161 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:43.161 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659140 2024-03-30T12:10:43.162 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:43.162 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:10:43.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659140 2024-03-30T12:10:43.163 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:43.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659140 2024-03-30T12:10:43.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-03-30T12:10:43.164 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 176093659140 2024-03-30T12:10:43.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:10:43.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659140 -lt 176093659140 2024-03-30T12:10:43.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:43.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-03-30T12:10:43.485 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:43.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:10:43.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-03-30T12:10:43.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:43.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-03-30T12:10:43.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-03-30T12:10:43.489 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 188978561027 2024-03-30T12:10:43.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:10:43.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561027 2024-03-30T12:10:43.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:10:43.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-206158430211 2024-03-30T12:10:43.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:10:43.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:10:43.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-206158430211 2024-03-30T12:10:43.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:10:43.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430211 2024-03-30T12:10:43.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-03-30T12:10:43.776 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 206158430211 2024-03-30T12:10:43.776 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:10:44.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430211 -lt 206158430211 2024-03-30T12:10:44.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:10:44.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:10:44.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:10:44.456 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:10:44.736 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T12:10:44.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:10:44.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:10:44.736 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:10:45.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T12:10:45.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:10:45.092 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:10:45.092 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-30T12:10:45.093 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-30T12:10:45.093 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-30T12:10:45.093 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-30T12:10:45.093 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:10:45.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-30T12:10:45.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-30T12:10:45.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-30T12:10:45.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-30T12:10:45.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:10:45.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:10:45.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:10:45.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:10:45.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:45.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-30T12:10:45.928 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-30T12:10:45.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:10:45.940 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:10:46.229 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:10:08.547980+0000 '>' 2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:46.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:10:47.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:10:47.231 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:10:47.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:10:47.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:10:47.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:10:47.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:10:47.232 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:10:47.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:10:08.547980+0000 '>' 2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:47.545 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:10:48.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:10:48.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:10:48.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:10:48.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:10:48.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:10:48.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:10:48.547 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:10:48.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:10:08.547980+0000 '>' 2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:48.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:10:49.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:10:49.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:10:49.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:10:49.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:10:49.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:10:49.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:10:49.841 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:10:50.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:10:46.532210+0000 '>' 2024-03-30T12:10:08.547980+0000 2024-03-30T12:10:50.133 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:10:50.133 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-30T12:10:50.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-30T12:10:50.134 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-30T12:10:50.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-30T12:10:50.134 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-30T12:10:50.135 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-30T12:10:50.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-30T12:10:50.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-30T12:10:50.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-30T12:10:50.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T12:10:50.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:10:50.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:10:50.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:10:50.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:10:50.244 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:10:50.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:10:50.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:10:50.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:10:50.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:10:50.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:10:50.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:10:50.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:10:50.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:10:50.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:10:50.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:10:50.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:10:50.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:10:50.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:10:50.356 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:10:50.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:10:50.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:10:50.357 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:10:50.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:10:50.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:50.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:50.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:10:50.376 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:10:50.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:10:50.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:10:50.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:10:50.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:10:50.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:10:50.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:10:50.379 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:10:50.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:10:50.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:10:50.381 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:10:50.381 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:10:50.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:10:50.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:10:50.382 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:10:50.383 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:10:50.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:10:50.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:10:50.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:10:50.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:50.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:50.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:10:50.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:10:50.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:10:50.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T12:10:50.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T12:10:50.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:50.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:50.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-30T12:10:50.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T12:10:50.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-30T12:10:50.808 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:50.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-30T12:10:51.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T12:10:51.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T12:10:51.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:10:51.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:10:51.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T12:10:51.011 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T12:10:51.011 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:10:51.011 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:10:51.011 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:10:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:10:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:51.012 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:10:51.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:10:51.012 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T12:10:51.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T12:10:51.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:10:51.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:10:51.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T12:10:51.097 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T12:10:51.098 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:10:51.098 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:10:51.098 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:10:51.098 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:10:51.099 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:51.099 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:51.099 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:10:51.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:10:51.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T12:10:51.179 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T12:10:51.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T12:10:51.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:10:51.995 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:10:51.995 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:10:51.995 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:51.996 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:51.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:51.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:10:51.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T12:10:52.028 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:10:52.029 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:10:52.030 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff 2024-03-30T12:10:52.031 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff 2024-03-30T12:10:52.032 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff' 2024-03-30T12:10:52.032 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:10:52.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMAQhmJLc4AxAAifkAK622CvRa+N1cGlVUmA== 2024-03-30T12:10:52.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMAQhmJLc4AxAAifkAK622CvRa+N1cGlVUmA=="}' 2024-03-30T12:10:52.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff -i td/osd-scrub-repair/0/new.json 2024-03-30T12:10:52.250 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:10:52.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T12:10:52.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBMAQhmJLc4AxAAifkAK622CvRa+N1cGlVUmA== --osd-uuid 0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff 2024-03-30T12:10:52.291 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.297+0000 7fc30620d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:52.301 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.307+0000 7fc30620d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:52.301 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.309+0000 7fc30620d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:52.302 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.309+0000 7fc30620d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:10:52.302 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.309+0000 7fc30620d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:10:52.302 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.309+0000 7fc30620d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:10:52.392 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:52.399+0000 7fc30620d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T12:10:57.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T12:10:57.994 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:10:57.995 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T12:10:57.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T12:10:57.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:10:58.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T12:10:58.386 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:10:58.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-30T12:10:58.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:10:58.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:10:58.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:10:58.414 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:58.422+0000 7f6af3824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:58.416 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:58.424+0000 7f6af3824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:58.417 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:58.425+0000 7f6af3824640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:10:58.669 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:10:58.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:58.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:10:58.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:10:58.940+0000 7f6af3824640 -1 Falling back to public interface 2024-03-30T12:10:58.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:10:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:10:59.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:10:59.953 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:10:59.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:10:59.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:10:59.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:00.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:01.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:01.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:01.249 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:11:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:11:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:01.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:01.266 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:01.274+0000 7f6af3824640 -1 osd.0 0 log_to_monitors true 2024-03-30T12:11:01.691 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:02.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:11:02.694 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:11:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:02.695 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:02.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:03.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:03.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:03.988 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:11:03.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:11:03.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:03.989 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:04.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:05.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:05.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:05.279 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:11:05.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:11:05.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:05.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:05.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:06.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:06.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:06.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:11:06.572 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:11:06.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:06.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:06.864 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:07.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:07.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:07.866 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:11:07.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:11:07.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:07.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:08.213 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:09.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:09.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:09.214 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:11:09.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:11:09.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:09.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:09.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:10.508 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:10.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:10.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:11:10.509 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:11:10.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:10.509 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:10.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:11.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:11.810 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:11:11.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:11:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:11.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:12.173 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:13.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:13.174 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:11:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:11:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:13.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:13.625 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:14.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:14.626 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:14.627 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:11:14.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:11:14.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:14.627 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:15.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:16.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:16.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:16.284 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:11:16.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:11:16.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:16.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:16.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:17.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:17.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:17.919 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:11:17.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:11:17.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:17.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:18.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:19.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:19.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:19.915 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:11:19.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:11:19.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:19.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:20.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:21.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:21.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:21.386 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:11:21.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:11:21.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:21.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:21.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:22.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:22.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:22.928 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:11:22.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:11:22.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:22.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:11:23.264 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1110079855,v1:127.0.0.1:6803/1110079855] [v2:127.0.0.1:6804/1110079855,v1:127.0.0.1:6805/1110079855] exists,up 0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:11:23.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:11:23.270 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:11:23.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63338b88-ff66-432e-971a-f5c6dd2295c0 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 63338b88-ff66-432e-971a-f5c6dd2295c0' 2024-03-30T12:11:23.272 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 63338b88-ff66-432e-971a-f5c6dd2295c0 2024-03-30T12:11:23.273 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:11:23.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrAQhmSh2GERAAj7VjT1HK1Ny97/eO3RpjWg== 2024-03-30T12:11:23.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrAQhmSh2GERAAj7VjT1HK1Ny97/eO3RpjWg=="}' 2024-03-30T12:11:23.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63338b88-ff66-432e-971a-f5c6dd2295c0 -i td/osd-scrub-repair/1/new.json 2024-03-30T12:11:23.661 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:11:23.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T12:11:23.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBrAQhmSh2GERAAj7VjT1HK1Ny97/eO3RpjWg== --osd-uuid 63338b88-ff66-432e-971a-f5c6dd2295c0 2024-03-30T12:11:23.703 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.711+0000 7f79fe011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:11:23.738 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.745+0000 7f79fe011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:11:23.738 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.747+0000 7f79fe011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:11:23.739 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.747+0000 7f79fe011640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:11:23.739 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.747+0000 7f79fe011640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:11:23.739 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.747+0000 7f79fe011640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:11:23.815 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:23.823+0000 7f79fe011640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T12:11:29.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T12:11:29.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:11:29.875 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T12:11:29.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T12:11:29.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:11:30.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T12:11:30.243 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:11:30.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-30T12:11:30.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:11:30.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:11:30.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:11:30.271 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:30.279+0000 7f84b1aa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:11:30.273 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:30.281+0000 7f84b1aa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:11:30.274 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:30.282+0000 7f84b1aa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:11:30.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T12:11:30.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:11:30.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:11:30.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:11:30.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:11:30.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:30.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:11:30.530 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:11:30.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:30.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:30.788 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:30.796+0000 7f84b1aa2640 -1 Falling back to public interface 2024-03-30T12:11:30.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:31.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:31.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:31.809 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:11:31.810 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:11:31.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:31.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:32.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:33.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:33.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:33.101 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:11:33.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:11:33.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:33.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:33.227 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:11:33.235+0000 7f84b1aa2640 -1 osd.1 0 log_to_monitors true 2024-03-30T12:11:33.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:34.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:34.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:34.489 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:11:34.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:11:34.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:34.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:34.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:35.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:35.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:35.781 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:11:35.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:11:35.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:35.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:36.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:37.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:37.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:37.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:11:37.091 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:11:37.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:37.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:37.428 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:38.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:38.430 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:38.430 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:11:38.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:11:38.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:38.431 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:38.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:39.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:39.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:39.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:11:39.724 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:11:39.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:39.725 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:40.039 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:41.040 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:41.040 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:11:41.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:41.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:11:41.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:41.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:41.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:42.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:42.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:42.363 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:11:42.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:11:42.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:42.363 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:42.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:43.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:43.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:43.659 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:11:43.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:11:43.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:43.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:43.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:44.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:44.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:44.954 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:11:44.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:11:44.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:44.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:45.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:46.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:46.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:46.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:11:46.250 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:11:46.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:46.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:46.794 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:47.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:47.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:47.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:11:47.796 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:11:47.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:47.797 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:48.332 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:49.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:49.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:49.334 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:11:49.335 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:11:49.336 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:49.336 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:50.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:51.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:51.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:51.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:11:51.086 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:11:51.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:51.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:51.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:52.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:52.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:52.378 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:11:52.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:11:52.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:52.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:53.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:54.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:54.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:54.131 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:11:54.132 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:11:54.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:54.132 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:54.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:55.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:55.573 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:11:55.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:55.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:11:55.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:55.574 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:55.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:11:56.867 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:11:56.867 INFO:tasks.workunit.client.0.smithi003.stdout:19 2024-03-30T12:11:56.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:11:56.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-30T12:11:56.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:11:56.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:11:57.189 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/609422189,v1:127.0.0.1:6811/609422189] [v2:127.0.0.1:6812/609422189,v1:127.0.0.1:6813/609422189] exists,up 63338b88-ff66-432e-971a-f5c6dd2295c0 2024-03-30T12:11:57.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:11:57.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:11:57.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:11:57.190 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-03-30T12:11:57.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-30T12:11:57.462 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' does not exist 2024-03-30T12:11:57.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-30T12:11:57.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-30T12:11:57.854 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'rbd' created 2024-03-30T12:11:57.865 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:11:58.866 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-30T12:12:00.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-30T12:12:00.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:12:00.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:12:00.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:12:00.883 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:12:00.883 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:12:00.883 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:12:00.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:12:00.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:12:00.884 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:12:00.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:12:00.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:12:00.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:12:00.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:12:00.985 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:12:00.985 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:12:01.264 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:12:01.264 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:12:01.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:12:01.265 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:01.265 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:12:01.369 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-03-30T12:12:01.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-03-30T12:12:01.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-03-30T12:12:01.370 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:01.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:12:01.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672963 2024-03-30T12:12:01.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672963 2024-03-30T12:12:01.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672963' 2024-03-30T12:12:01.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:01.475 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-03-30T12:12:01.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:01.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:12:01.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-03-30T12:12:01.477 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:01.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-03-30T12:12:01.478 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836495 2024-03-30T12:12:01.479 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-03-30T12:12:01.479 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:01.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-03-30T12:12:01.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:02.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:12:02.760 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:03.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-03-30T12:12:03.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:03.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672963 2024-03-30T12:12:03.057 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:03.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:12:03.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672963 2024-03-30T12:12:03.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:03.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672963 2024-03-30T12:12:03.060 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-03-30T12:12:03.060 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672963 2024-03-30T12:12:03.061 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:12:03.385 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672963 2024-03-30T12:12:03.385 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:12:03.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:12:03.386 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:12:03.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:12:04.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-30T12:12:04.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:12:04.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:12:04.017 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:12:04.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-30T12:12:04.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:12:04.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:12:04.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-03-30T12:12:04.384 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-03-30T12:12:04.759 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'psr_pool' created 2024-03-30T12:12:04.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:12:05.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:12:05.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:12:05.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:12:05.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:12:05.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:12:05.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:12:05.875 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:12:06.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:12:06.168 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:12:06.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:12:06.168 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:06.168 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:12:06.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-30T12:12:06.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-30T12:12:06.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-30T12:12:06.293 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:06.293 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:12:06.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-30T12:12:06.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-30T12:12:06.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672965' 2024-03-30T12:12:06.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:06.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-30T12:12:06.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:06.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:12:06.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-30T12:12:06.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:06.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-30T12:12:06.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-30T12:12:06.401 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836497 2024-03-30T12:12:06.402 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:06.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-30T12:12:06.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:07.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:12:07.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:07.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-30T12:12:07.979 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:08.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T12:12:08.980 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:09.276 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-30T12:12:09.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:09.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-30T12:12:09.277 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:09.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:12:09.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-30T12:12:09.278 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:09.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-30T12:12:09.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-30T12:12:09.280 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672965 2024-03-30T12:12:09.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:12:09.561 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672965 2024-03-30T12:12:09.561 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:12:09.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:12:09.562 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:12:09.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T12:12:09.911 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:12:09.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:12:09.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:12:09.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:12:09.912 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:12:09.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:12:09.913 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:12:10.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T12:12:10.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:12:10.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:12:10.192 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-03-30T12:12:10.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-03-30T12:12:10.922 INFO:tasks.workunit.client.0.smithi003.stderr:noscrub is unset 2024-03-30T12:12:10.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-03-30T12:12:11.441 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is unset 2024-03-30T12:12:11.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-30T12:12:11.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-30T12:12:11.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-03-30T12:12:11.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-03-30T12:12:11.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-03-30T12:12:11.712 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-03-30T12:12:11.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-03-30T12:12:11.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T12:12:11.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-03-30T12:12:11.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-03-30T12:12:11.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-03-30T12:12:11.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-03-30T12:12:11.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-03-30T12:12:11.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:12:12.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-03-30T12:12:12.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-03-30T12:12:12.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-03-30T12:12:12.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T12:12:12.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-30T12:12:12.284 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:12:12.285 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:12:12.591 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:12:12.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-30T12:12:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-30T12:12:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:12:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:12:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:12:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:12:14.529 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:12:14.530 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-30T12:12:14.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:12:14.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:12:14.532 INFO:tasks.workunit.client.0.smithi003.stderr:start osd.0 2024-03-30T12:12:14.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-30T12:12:14.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:12:14.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:12:14.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:12:14.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:12:14.533 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:12:14.561 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:14.569+0000 7fd586a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:14.564 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:14.572+0000 7fd586a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:14.565 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:14.573+0000 7fd586a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:12:14.827 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:12:14.828 INFO:tasks.workunit.client.0.smithi003.stderr:0 2024-03-30T12:12:14.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:12:14.828 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:12:15.080 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:15.088+0000 7fd586a1b640 -1 Falling back to public interface 2024-03-30T12:12:15.123 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:12:16.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:12:16.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:12:16.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:12:16.125 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:12:16.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:12:16.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:12:16.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:12:17.416 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:12:17.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:12:17.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:12:17.417 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-03-30T12:12:17.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:12:17.417 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:12:17.523 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:17.531+0000 7fd586a1b640 -1 osd.0 25 log_to_monitors true 2024-03-30T12:12:17.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:12:18.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:12:18.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:12:18.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:12:18.747 INFO:tasks.workunit.client.0.smithi003.stderr:3 2024-03-30T12:12:18.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:12:18.747 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:12:19.095 INFO:tasks.workunit.client.0.smithi003.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/2780455618,v1:127.0.0.1:6803/2780455618] [v2:127.0.0.1:6804/2780455618,v1:127.0.0.1:6805/2780455618] exists,up 0e2b4963-20d9-4fa7-87a6-1f1dab7dbcff 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:12:19.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:12:19.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:12:19.099 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:12:19.099 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:12:19.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:12:19.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:12:19.099 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:12:19.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:12:19.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:12:19.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:12:19.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:12:19.185 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:12:19.185 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:12:19.499 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:12:19.500 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:12:19.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:12:19.500 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:19.500 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:12:19.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-30T12:12:19.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-30T12:12:19.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-30T12:12:19.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:19.605 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:12:19.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-30T12:12:19.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-30T12:12:19.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672969' 2024-03-30T12:12:19.710 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:19.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-30T12:12:19.714 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:19.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:12:19.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-30T12:12:19.715 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:19.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-30T12:12:19.715 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-30T12:12:19.715 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.0 seq 124554051586 2024-03-30T12:12:19.716 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:19.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-30T12:12:19.995 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:20.996 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:12:20.996 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:21.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-30T12:12:21.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:21.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-30T12:12:21.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:21.316 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:12:21.317 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-30T12:12:21.317 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:21.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-30T12:12:21.318 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-30T12:12:21.318 INFO:tasks.workunit.client.0.smithi003.stderr:waiting osd.1 seq 42949672969 2024-03-30T12:12:21.318 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:12:21.601 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-03-30T12:12:21.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:12:21.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:12:21.601 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:12:21.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-30T12:12:21.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:12:21.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:12:21.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:12:21.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:12:21.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:12:21.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:12:21.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:12:22.230 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-30T12:12:22.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:12:22.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:12:22.231 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:12:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-30T12:12:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:12:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:12:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-03-30T12:12:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-30T12:12:22.587 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-03-30T12:12:22.611 INFO:tasks.workunit.client.0.smithi003.stderr:No scrub information available for pg 2.0 2024-03-30T12:12:22.611 INFO:tasks.workunit.client.0.smithi003.stderr:error 2: (2) No such file or directory 2024-03-30T12:12:22.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-03-30T12:12:22.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-03-30T12:12:22.613 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-03-30T12:12:22.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-30T12:12:22.613 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:22.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:12:22.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:22.614 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-30T12:12:22.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:22.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-03-30T12:12:23.183 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-03-30T12:12:23.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-03-30T12:12:04.721152+0000 last_deep_scrub_stamp 2024-03-30T12:12:23.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T12:12:23.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:23.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:23.197 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-30T12:12:23.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:04.721152+0000 '>' 2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:23.474 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:24.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:24.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:24.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-30T12:12:24.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:24.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:12:24.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:24.476 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-30T12:12:24.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:04.721152+0000 '>' 2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:24.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:25.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:25.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:25.770 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-30T12:12:25.770 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:25.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:12:25.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:25.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-30T12:12:26.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:04.721152+0000 '>' 2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:26.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:27.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:27.062 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:27.063 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-30T12:12:27.063 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:27.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:12:27.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:27.064 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-30T12:12:27.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:04.721152+0000 '>' 2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:27.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:28.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:28.387 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:28.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-30T12:12:28.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:28.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-30T12:12:28.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:28.388 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-30T12:12:28.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:23.646893+0000 '>' 2024-03-30T12:12:04.721152+0000 2024-03-30T12:12:28.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:12:28.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-30T12:12:28.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-03-30T12:12:28.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-30T12:12:28.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-30T12:12:28.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:12:28.704 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:12:28.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:12:28.984 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:12:28.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:12:28.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:28.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:12:29.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-30T12:12:29.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-30T12:12:29.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-30T12:12:29.090 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:29.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:12:29.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-30T12:12:29.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-30T12:12:29.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672972' 2024-03-30T12:12:29.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:29.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-30T12:12:29.196 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:29.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:12:29.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-30T12:12:29.198 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:29.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-30T12:12:29.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-30T12:12:29.200 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 124554051589 2024-03-30T12:12:29.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:29.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-30T12:12:29.482 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:30.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:12:30.484 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:30.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-03-30T12:12:30.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:30.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-30T12:12:30.779 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:30.780 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:12:30.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-30T12:12:30.780 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:30.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-30T12:12:30.782 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-30T12:12:30.782 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672972 2024-03-30T12:12:30.782 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:12:31.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-30T12:12:31.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-03-30T12:12:31.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:31.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:31.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:31.125 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:31.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-03-30T12:12:23.646893+0000 2024-03-30T12:12:31.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-30T12:12:31.506 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:12:31.506 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": false, 2024-03-30T12:12:31.506 INFO:tasks.workunit.client.0.smithi003.stdout: "must": false, 2024-03-30T12:12:31.506 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "2024-03-23T12:10:51.514917+0000" 2024-03-30T12:12:31.506 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-03-30T12:12:23.646893+0000 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:12:23.646893+0000 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:31.516 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:23.646893+0000 '>' 2024-03-30T12:12:23.646893+0000 2024-03-30T12:12:31.795 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:32.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:32.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:32.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:12:32.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:32.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:32.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:32.797 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:33.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:23.646893+0000 '>' 2024-03-30T12:12:23.646893+0000 2024-03-30T12:12:33.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:34.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:34.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:34.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:12:34.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:34.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:34.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:34.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:34.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:31.745984+0000 '>' 2024-03-30T12:12:23.646893+0000 2024-03-30T12:12:34.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:12:34.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-03-30T12:12:34.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-30T12:12:34.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-03-30T12:12:34.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-30T12:12:34.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-03-30T12:12:34.800 INFO:tasks.workunit.client.0.smithi003.stderr:nodeep-scrub is set 2024-03-30T12:12:34.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-03-30T12:12:34.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-30T12:12:34.919 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:12:34.919 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:12:35.214 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:12:35.215 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:12:35.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:12:35.215 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:35.215 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:12:35.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-03-30T12:12:35.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-03-30T12:12:35.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-03-30T12:12:35.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:35.320 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:12:35.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-30T12:12:35.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-30T12:12:35.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-42949672974' 2024-03-30T12:12:35.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:35.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-03-30T12:12:35.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:35.427 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:12:35.427 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-03-30T12:12:35.428 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:35.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-03-30T12:12:35.429 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-03-30T12:12:35.429 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 124554051591 2024-03-30T12:12:35.429 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:35.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-03-30T12:12:35.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:36.713 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:12:36.713 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:37.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-03-30T12:12:37.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:37.006 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-30T12:12:37.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:37.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:12:37.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-30T12:12:37.008 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:37.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-30T12:12:37.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-30T12:12:37.009 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672974 2024-03-30T12:12:37.010 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:12:37.289 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-03-30T12:12:37.290 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-03-30T12:12:42.291 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-30T12:12:42.400 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:12:42.400 INFO:tasks.workunit.client.0.smithi003.stdout: "deep": false, 2024-03-30T12:12:42.400 INFO:tasks.workunit.client.0.smithi003.stdout: "must": false, 2024-03-30T12:12:42.400 INFO:tasks.workunit.client.0.smithi003.stdout: "stamp": "2024-03-23T12:11:02.408836+0000" 2024-03-30T12:12:42.400 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:12:42.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-03-30T12:12:42.410 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-03-30T12:12:42.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-03-30T12:12:42.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-03-30T12:12:43.412 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-03-30T12:12:43.414 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-03-30T12:12:43.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-03-30T12:12:43.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-03-30T12:12:43.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-30T12:12:43.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-03-30T12:12:43.415 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-30T12:12:43.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-30T12:12:43.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:12:43.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:12:43.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:12:43.737 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:12:43.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:12:43.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:43.737 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:12:43.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051594 2024-03-30T12:12:43.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051594 2024-03-30T12:12:43.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594' 2024-03-30T12:12:43.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:12:43.844 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:12:43.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-30T12:12:43.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-30T12:12:43.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-42949672977' 2024-03-30T12:12:43.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:43.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051594 2024-03-30T12:12:43.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:43.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:12:43.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051594 2024-03-30T12:12:43.951 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:43.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051594 2024-03-30T12:12:43.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-03-30T12:12:43.952 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 124554051594 2024-03-30T12:12:43.952 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:44.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051594 2024-03-30T12:12:44.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:12:45.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:12:45.234 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:12:45.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051594 -lt 124554051594 2024-03-30T12:12:45.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:12:45.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-30T12:12:45.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:12:45.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:12:45.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-30T12:12:45.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:12:45.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-30T12:12:45.530 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672977 2024-03-30T12:12:45.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-30T12:12:45.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:12:45.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-30T12:12:45.814 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-03-30T12:12:45.815 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-03-30T12:12:45.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-03-30T12:12:45.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:45.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:45.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:45.815 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:46.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-03-23T12:11:02.408836+0000 2024-03-30T12:12:46.091 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-03-30T12:12:46.361 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 2.0 on osd.0 to scrub 2024-03-30T12:12:46.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-03-23T12:11:02.408836+0000 2024-03-30T12:12:46.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-23T12:11:02.408836+0000 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:46.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:46.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-23T12:11:02.408836+0000 '>' 2024-03-23T12:11:02.408836+0000 2024-03-30T12:12:46.652 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:47.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:47.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:47.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:12:47.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:47.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:47.654 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:47.655 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:47.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-23T12:11:02.408836+0000 '>' 2024-03-23T12:11:02.408836+0000 2024-03-30T12:12:47.944 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:12:48.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:12:48.947 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:12:48.947 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-30T12:12:48.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-30T12:12:48.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:12:48.948 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:12:48.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-30T12:12:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:12:46.944026+0000 '>' 2024-03-23T12:11:02.408836+0000 2024-03-30T12:12:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:12:49.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-03-30T12:12:49.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-30T12:12:49.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-03-30T12:12:49.239 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-03-30T12:12:49.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T12:12:49.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:12:49.266 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:12:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:12:49.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:12:49.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:12:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:12:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:12:49.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:12:49.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:12:49.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:12:49.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:12:49.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:12:49.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:12:49.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:12:49.377 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:12:49.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:12:49.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:12:49.378 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:12:49.378 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:12:49.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:12:49.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:12:49.380 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:12:49.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:12:49.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:49.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:49.392 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:12:49.393 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:12:49.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:12:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:12:49.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:12:49.397 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:12:49.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:12:49.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:12:49.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:12:49.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:12:49.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:12:49.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:12:49.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:12:49.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:12:49.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:12:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:12:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:12:49.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:12:49.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:49.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:49.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:12:49.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:12:49.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:12:49.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T12:12:49.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T12:12:49.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:49.405 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:49.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T12:12:49.408 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T12:12:49.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T12:12:49.408 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T12:12:49.409 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T12:12:49.777 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:12:49.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T12:12:49.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T12:12:49.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T12:12:49.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:12:49.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:12:49.987 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:49.988 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:49.989 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:12:49.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:12:49.989 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:12:50.073 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:12:50.074 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:12:50.074 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:50.074 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:50.074 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:12:50.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:12:50.074 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T12:12:50.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-03-30T12:12:50.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T12:12:50.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T12:12:50.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T12:12:50.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T12:12:50.155 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T12:12:50.156 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T12:12:50.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:12:50.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:12:50.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:12:50.991 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:12:50.992 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-03-30T12:12:50.993 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-03-30T12:12:50.997 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:12:50.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:12:50.999 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:12:51.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:12:51.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=21173673-8178-4e5a-8c6c-dcf1512a8f9a 2024-03-30T12:12:51.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 21173673-8178-4e5a-8c6c-dcf1512a8f9a' 2024-03-30T12:12:51.001 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 21173673-8178-4e5a-8c6c-dcf1512a8f9a 2024-03-30T12:12:51.001 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:12:51.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDAQhmGQxzARAAA+NSBLMuQ6lHAQQFJyzpdQ== 2024-03-30T12:12:51.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDAQhmGQxzARAAA+NSBLMuQ6lHAQQFJyzpdQ=="}' 2024-03-30T12:12:51.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 21173673-8178-4e5a-8c6c-dcf1512a8f9a -i td/osd-scrub-repair/0/new.json 2024-03-30T12:12:51.227 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:12:51.237 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T12:12:51.238 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDDAQhmGQxzARAAA+NSBLMuQ6lHAQQFJyzpdQ== --osd-uuid 21173673-8178-4e5a-8c6c-dcf1512a8f9a 2024-03-30T12:12:51.268 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.276+0000 7ff82ce89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:51.278 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.286+0000 7ff82ce89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:51.279 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.287+0000 7ff82ce89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:51.279 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.287+0000 7ff82ce89640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:12:51.280 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.287+0000 7ff82ce89640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:12:51.280 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.287+0000 7ff82ce89640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:12:51.378 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:51.386+0000 7ff82ce89640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T12:12:56.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T12:12:56.936 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:12:56.937 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T12:12:56.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T12:12:56.937 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:12:57.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T12:12:57.348 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:12:57.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:12:57.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:12:57.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:12:57.349 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:12:57.375 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:57.383+0000 7f2f60181640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:57.377 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:57.385+0000 7f2f60181640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:57.378 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:57.386+0000 7f2f60181640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:12:57.631 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:12:57.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:12:57.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:12:57.632 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:12:57.896 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:12:57.904+0000 7f2f60181640 -1 Falling back to public interface 2024-03-30T12:12:57.914 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:12:58.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:12:58.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:12:58.916 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:12:58.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:12:58.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:12:58.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:12:59.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:00.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:00.211 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:00.212 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:13:00.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:13:00.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:00.212 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:00.228 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:00.235+0000 7f2f60181640 -1 osd.0 0 log_to_monitors true 2024-03-30T12:13:00.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:01.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:01.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:01.658 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:13:01.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:13:01.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:01.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:01.948 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:02.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:02.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:02.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:13:02.950 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:13:02.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:02.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:03.242 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:04.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:04.243 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:04.243 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:13:04.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:13:04.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:04.244 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:04.537 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:05.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:05.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:05.539 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:13:05.539 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:13:05.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:05.540 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:05.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:06.952 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:06.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:07.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:08.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:08.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:08.249 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:13:08.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:13:08.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:08.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:08.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:09.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:09.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:09.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:13:09.548 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:13:09.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:09.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:09.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:10.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:10.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:10.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:13:10.960 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:13:10.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:10.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:11.250 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:12.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:12.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:12.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:13:12.252 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:13:12.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:12.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:12.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:13.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:13.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:13.800 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:13:13.801 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:13:13.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:13.801 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:14.379 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:15.376 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:15.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:15.377 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:13:15.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:13:15.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:15.377 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:16.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:17.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:17.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:17.254 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:13:17.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:13:17.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:17.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:18.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:19.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:19.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:19.194 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:13:19.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:13:19.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:19.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:19.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:20.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:20.642 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:20.642 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:13:20.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:13:20.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:20.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:21.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:22.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:22.149 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:22.150 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:13:22.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:13:22.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:22.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:13:22.512 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1201584732,v1:127.0.0.1:6803/1201584732] [v2:127.0.0.1:6804/1201584732,v1:127.0.0.1:6805/1201584732] exists,up 21173673-8178-4e5a-8c6c-dcf1512a8f9a 2024-03-30T12:13:22.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:13:22.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:13:22.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:13:22.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:13:22.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:13:22.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:13:22.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=afbbaa98-cb21-46ab-b553-d6ed82799915 2024-03-30T12:13:22.520 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 afbbaa98-cb21-46ab-b553-d6ed82799915 2024-03-30T12:13:22.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 afbbaa98-cb21-46ab-b553-d6ed82799915' 2024-03-30T12:13:22.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:13:22.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDiAQhm08tXIBAAQCOLRousrJEDIec1cd8p4w== 2024-03-30T12:13:22.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDiAQhm08tXIBAAQCOLRousrJEDIec1cd8p4w=="}' 2024-03-30T12:13:22.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new afbbaa98-cb21-46ab-b553-d6ed82799915 -i td/osd-scrub-repair/1/new.json 2024-03-30T12:13:22.870 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:13:22.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T12:13:22.882 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDiAQhm08tXIBAAQCOLRousrJEDIec1cd8p4w== --osd-uuid afbbaa98-cb21-46ab-b553-d6ed82799915 2024-03-30T12:13:22.912 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:22.919+0000 7f6fdbe1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:13:22.955 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:22.962+0000 7f6fdbe1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:13:22.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:22.964+0000 7f6fdbe1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:13:22.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:22.964+0000 7f6fdbe1f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:13:22.956 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:22.964+0000 7f6fdbe1f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:13:22.957 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:22.964+0000 7f6fdbe1f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:13:23.030 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:23.038+0000 7f6fdbe1f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T12:13:29.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T12:13:29.083 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:13:29.084 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T12:13:29.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T12:13:29.084 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:13:29.453 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:13:29.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T12:13:29.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:13:29.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:13:29.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:13:29.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:13:29.480 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:29.488+0000 7f675a306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:13:29.482 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:29.490+0000 7f675a306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:13:29.483 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:29.491+0000 7f675a306640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:13:29.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T12:13:29.737 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:13:29.738 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:29.739 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:30.015 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:30.023+0000 7f675a306640 -1 Falling back to public interface 2024-03-30T12:13:30.016 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:31.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:31.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:31.018 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:13:31.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:13:31.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:31.019 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:31.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:32.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:32.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:32.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:13:32.315 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:13:32.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:32.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:32.445 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:13:32.453+0000 7f675a306640 -1 osd.1 0 log_to_monitors true 2024-03-30T12:13:32.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:33.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:33.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:33.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:13:33.638 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:13:33.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:33.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:33.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:34.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:34.934 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:34.934 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:13:34.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:13:34.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:34.935 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:35.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:36.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:36.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:36.262 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:13:36.263 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:13:36.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:36.263 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:36.653 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:37.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:37.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:37.655 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:13:37.655 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:13:37.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:37.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:37.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:38.950 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:38.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:38.951 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:13:38.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:13:38.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:38.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:39.246 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:40.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:40.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:40.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:13:40.248 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:13:40.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:40.249 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:40.553 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:41.554 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:41.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:41.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:13:41.555 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:13:41.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:41.555 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:41.846 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:42.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:42.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:42.847 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:13:42.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:13:42.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:42.848 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:43.304 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:44.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:44.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:44.305 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:13:44.306 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:13:44.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:44.306 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:44.716 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:45.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:45.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:45.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:13:45.719 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:13:45.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:45.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:46.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:47.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:47.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:47.488 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:13:47.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:13:47.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:47.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:48.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:49.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:49.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:49.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:13:49.273 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:13:49.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:49.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:49.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:50.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:50.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:50.639 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:13:50.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:13:50.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:50.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:51.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:52.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:52.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:52.303 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:13:52.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:13:52.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:52.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:53.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:54.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:54.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:54.014 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:13:54.014 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:13:54.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:54.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:54.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:13:55.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:13:55.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:13:55.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:13:55.618 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:13:55.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:13:55.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:13:55.915 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/610582716,v1:127.0.0.1:6811/610582716] [v2:127.0.0.1:6812/610582716,v1:127.0.0.1:6813/610582716] exists,up afbbaa98-cb21-46ab-b553-d6ed82799915 2024-03-30T12:13:55.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:13:55.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:13:55.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:13:55.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-03-30T12:13:55.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-30T12:13:56.435 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T12:13:56.447 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:13:57.448 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-03-30T12:13:57.948 INFO:tasks.workunit.client.0.smithi003.stderr:set pool 1 size to 2 2024-03-30T12:13:57.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-03-30T12:13:57.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:13:57.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:13:57.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:13:57.960 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:13:57.961 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:13:57.961 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:13:57.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:13:57.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:13:57.961 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:13:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:13:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:13:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:13:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:13:58.051 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:13:58.052 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:13:58.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:13:58.531 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:13:58.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:13:58.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:13:58.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:13:58.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-30T12:13:58.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-30T12:13:58.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-30T12:13:58.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:13:58.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:13:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-30T12:13:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-30T12:13:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-03-30T12:13:58.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:13:58.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-30T12:13:58.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:13:58.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:13:58.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-30T12:13:58.774 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:13:58.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-30T12:13:58.775 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836493 2024-03-30T12:13:58.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-30T12:13:58.776 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:13:59.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836493 2024-03-30T12:13:59.057 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:14:00.058 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:14:00.058 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:00.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-30T12:14:00.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:00.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-30T12:14:00.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:00.356 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:14:00.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-30T12:14:00.357 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:00.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-30T12:14:00.358 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672965 2024-03-30T12:14:00.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-30T12:14:00.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:14:00.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-30T12:14:00.681 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:14:00.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:14:00.682 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:14:01.041 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:14:01.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:14:01.319 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T12:14:01.320 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:14:01.320 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:14:01.320 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:14:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T12:14:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:14:01.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:14:01.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-03-30T12:14:01.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-03-30T12:14:01.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-03-30T12:14:01.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:01.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:01.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:01.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:02.890 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:02.891 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.259 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.260 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.401 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.538 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.779 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:03.896 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.189 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.432 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.549 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.783 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:04.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:04.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:05.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:05.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:05.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:05.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:05.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:05.286 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:05.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-30T12:14:05.403 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:14:05.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T12:14:05.522 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T12:14:05.804 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T12:14:05.804 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-30T12:14:05.804 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T12:14:06.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-03-30T12:14:06.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-03-30T12:14:06.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T12:14:06.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-30T12:14:06.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-30T12:14:06.090 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:14:06.373 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-03-30T12:14:06.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-03-30T12:14:06.374 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T12:14:06.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T12:14:06.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T12:14:06.375 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T12:14:06.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-03-30T12:14:06.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:14:06.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:14:06.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:14:06.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:14:06.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:14:06.660 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:14:06.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:14:06.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:14:06.967 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:14:06.967 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:14:06.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:14:06.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:14:06.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:14:07.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:14:07.274 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-03-30T12:14:07.275 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:07.276 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-03-30T12:14:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:07.277 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-30T12:14:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:07.278 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-30T12:14:08.440 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-30T12:14:08.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:08.954 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-03-30T12:14:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-30T12:14:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:08.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:08.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:08.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-30T12:14:10.017 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:104778fc:::obj2:head# 2024-03-30T12:14:10.532 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:10.533 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-03-30T12:14:10.534 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:10.535 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-30T12:14:11.330 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:8dd16f86:::obj3:head# 2024-03-30T12:14:11.845 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:11.845 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:11.847 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-30T12:14:12.608 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:0ee9ae15:::obj4:head# 2024-03-30T12:14:13.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:13.122 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-03-30T12:14:13.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:13.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-30T12:14:13.124 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:13.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:13.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:13.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:13.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:13.125 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-30T12:14:13.818 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:c52c9666:::obj5:head# 2024-03-30T12:14:14.336 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:14.336 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-03-30T12:14:14.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:14.338 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-30T12:14:14.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:14.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:14.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:14.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:14.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:14.339 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-30T12:14:14.995 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:233a42c1:::obj6:head# 2024-03-30T12:14:15.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:15.510 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-03-30T12:14:15.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:15.513 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-30T12:14:16.240 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:9a09113a:::obj7:head# 2024-03-30T12:14:16.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:16.754 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-03-30T12:14:16.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:16.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-30T12:14:16.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:16.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:16.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:16.756 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:16.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:16.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-30T12:14:17.450 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:a557efb1:::obj8:head# 2024-03-30T12:14:17.964 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:17.964 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-03-30T12:14:17.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:17.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-30T12:14:17.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:17.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:17.966 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:17.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:17.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:17.967 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-30T12:14:18.594 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:fe351e27:::obj9:head# 2024-03-30T12:14:19.110 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:19.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:19.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-30T12:14:19.738 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:d1337354:::obj10:head# 2024-03-30T12:14:20.252 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:20.252 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-03-30T12:14:20.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:20.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-30T12:14:20.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:20.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:20.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:20.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:20.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:20.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-30T12:14:20.914 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:eeae2a94:::obj11:head# 2024-03-30T12:14:21.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:21.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:21.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-30T12:14:22.451 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:af5f95cb:::obj12:head# 2024-03-30T12:14:23.216 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:23.217 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-03-30T12:14:23.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:23.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-30T12:14:23.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:23.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:23.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:23.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:23.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:23.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-30T12:14:23.854 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:a61ad63e:::obj13:head# 2024-03-30T12:14:24.372 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:24.372 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-03-30T12:14:24.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:24.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-30T12:14:24.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:24.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:24.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:24.374 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:24.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:24.375 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-30T12:14:24.989 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:e0a44829:::obj14:head# 2024-03-30T12:14:25.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:25.505 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:25.507 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-30T12:14:26.140 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:ab946124:::obj15:head# 2024-03-30T12:14:26.659 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:26.659 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-03-30T12:14:26.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:26.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-30T12:14:26.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:26.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:26.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:26.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:26.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:26.662 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-30T12:14:27.301 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:461f8b5e:::obj16:head# 2024-03-30T12:14:27.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:27.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-03-30T12:14:27.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:27.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-30T12:14:27.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:27.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:27.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:27.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:27.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:27.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-30T12:14:28.470 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:24fc6e92:::obj17:head# 2024-03-30T12:14:28.984 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:28.984 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-03-30T12:14:28.986 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:28.987 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-30T12:14:29.621 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:b9836a99:::obj18:head# 2024-03-30T12:14:30.139 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:30.140 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-03-30T12:14:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-30T12:14:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-30T12:14:30.141 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:14:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:30.142 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-30T12:14:30.765 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:00b31b6c:::obj19:head# 2024-03-30T12:14:31.280 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-30T12:14:31.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-03-30T12:14:31.282 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:31.283 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-30T12:14:31.917 INFO:tasks.workunit.client.0.smithi003.stdout:remove #1:24b5611b:::obj20:head# 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:14:32.433 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:32.434 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:32.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:14:32.436 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:14:32.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:14:32.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:14:32.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:14:32.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:14:32.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:14:32.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:14:32.438 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:14:32.466 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:32.473+0000 7f582bc21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:32.468 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:32.476+0000 7f582bc21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:32.470 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:32.477+0000 7f582bc21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:32.733 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:14:32.734 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:14:32.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:32.734 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:14:32.985 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:32.992+0000 7f582bc21640 -1 Falling back to public interface 2024-03-30T12:14:33.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:14:34.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:14:34.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:34.017 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:14:34.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:14:34.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:34.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:14:34.309 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:14:35.271 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:35.278+0000 7f582bc21640 -1 osd.1 17 log_to_monitors true 2024-03-30T12:14:35.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:14:35.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:35.310 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:14:35.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:14:35.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:35.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:14:35.605 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:14:36.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:14:36.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:36.607 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:14:36.607 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:14:36.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:36.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:14:36.901 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/731309983,v1:127.0.0.1:6803/731309983] [v2:127.0.0.1:6804/731309983,v1:127.0.0.1:6805/731309983] exists,up afbbaa98-cb21-46ab-b553-d6ed82799915 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:14:36.905 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:14:36.906 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:14:36.907 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:14:36.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:14:36.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:14:36.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:14:36.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:14:36.908 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:14:36.908 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:14:36.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:14:36.909 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:14:36.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:14:36.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:14:36.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:14:36.909 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:14:36.931 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:36.939+0000 7f52dae11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:36.932 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:36.941+0000 7f52dae11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:36.933 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:36.942+0000 7f52dae11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:37.196 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:14:37.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:14:37.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:37.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:14:37.448 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:37.456+0000 7f52dae11640 -1 Falling back to public interface 2024-03-30T12:14:37.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:14:38.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:14:38.477 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:38.478 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:14:38.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:14:38.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:38.478 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:14:38.767 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:14:39.705 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:39.713+0000 7f52dae11640 -1 osd.0 16 log_to_monitors true 2024-03-30T12:14:39.768 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:14:39.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:14:39.769 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:14:39.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:14:39.769 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:14:39.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:14:40.100 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/3014013647,v1:127.0.0.1:6811/3014013647] [v2:127.0.0.1:6812/3014013647,v1:127.0.0.1:6813/3014013647] exists,up 21173673-8178-4e5a-8c6c-dcf1512a8f9a 2024-03-30T12:14:40.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:14:40.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:14:40.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:14:40.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:14:40.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:14:40.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:14:40.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:14:40.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:14:40.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:14:40.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:14:40.193 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:14:40.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:14:40.487 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:14:40.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:14:40.488 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:14:40.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:14:40.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-03-30T12:14:40.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-03-30T12:14:40.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-03-30T12:14:40.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:14:40.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:14:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345923 2024-03-30T12:14:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345923 2024-03-30T12:14:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-03-30T12:14:40.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:40.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-03-30T12:14:40.772 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:40.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:14:40.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-03-30T12:14:40.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:40.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-03-30T12:14:40.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-03-30T12:14:40.775 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 98784247810 2024-03-30T12:14:40.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:41.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 98784247810 2024-03-30T12:14:41.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:14:42.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:14:42.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-03-30T12:14:42.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:42.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345923 2024-03-30T12:14:42.351 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:42.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:14:42.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345923 2024-03-30T12:14:42.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:42.354 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345923 2024-03-30T12:14:42.354 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 85899345923 2024-03-30T12:14:42.355 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-03-30T12:14:42.355 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:14:42.636 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345923 2024-03-30T12:14:42.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:14:42.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:14:42.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:14:43.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T12:14:43.130 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:14:43.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:14:43.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:14:43.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:14:43.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:14:43.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:14:43.131 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:14:43.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T12:14:43.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:14:43.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:14:43.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:14:43.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T12:14:43.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:14:43.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:14:43.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-03-30T12:14:43.757 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-30T12:14:43.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-30T12:14:43.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:14:43.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:14:43.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:14:43.758 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:14:44.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T12:13:56.326780+0000 2024-03-30T12:14:44.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-30T12:14:44.346 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-30T12:13:56.326780+0000 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:13:56.326780+0000 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:14:44.358 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:14:44.359 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:14:44.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:13:56.326780+0000 '>' 2024-03-30T12:13:56.326780+0000 2024-03-30T12:14:44.634 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:14:45.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:14:45.635 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:14:45.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:14:45.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:14:45.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:14:45.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:14:45.636 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:14:45.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:13:56.326780+0000 '>' 2024-03-30T12:13:56.326780+0000 2024-03-30T12:14:45.928 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:14:46.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:14:46.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:14:46.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:14:46.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:14:46.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:14:46.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:14:46.930 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:14:47.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:14:45.338144+0000 '>' 2024-03-30T12:13:56.326780+0000 2024-03-30T12:14:47.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:14:47.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-03-30T12:14:47.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:14:47.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:14:47.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:14:47.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:14:47.221 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:14:47.221 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:14:47.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:14:47.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:14:47.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:14:47.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:14:47.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:14:47.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:14:47.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:14:47.310 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:14:47.310 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:14:47.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:14:47.589 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:14:47.589 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:14:47.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:14:47.590 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:14:47.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247812 2024-03-30T12:14:47.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247812 2024-03-30T12:14:47.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812' 2024-03-30T12:14:47.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:14:47.696 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:14:47.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-03-30T12:14:47.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-03-30T12:14:47.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812 1-85899345925' 2024-03-30T12:14:47.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:47.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247812 2024-03-30T12:14:47.803 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:47.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:14:47.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247812 2024-03-30T12:14:47.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:47.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247812 2024-03-30T12:14:47.806 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 98784247812 2024-03-30T12:14:47.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2024-03-30T12:14:47.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:48.098 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247811 -lt 98784247812 2024-03-30T12:14:48.099 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:14:49.100 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:14:49.100 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:49.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247812 -lt 98784247812 2024-03-30T12:14:49.399 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:49.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345925 2024-03-30T12:14:49.400 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:49.400 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:14:49.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345925 2024-03-30T12:14:49.401 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-03-30T12:14:49.402 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2024-03-30T12:14:49.403 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 85899345925 2024-03-30T12:14:49.403 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:14:49.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345925 2024-03-30T12:14:49.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:14:49.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:14:49.684 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:14:50.059 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:14:50.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T12:14:50.353 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:14:50.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:14:50.354 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:14:50.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T12:14:50.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:14:50.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:14:50.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-03-30T12:14:50.971 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T12:14:50.971 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-30T12:14:46.085696+0000 16'30 24:118 [1,0] 1 [1,0] 1 16'30 2024-03-30T12:14:45.338144+0000 16'30 2024-03-30T12:14:45.338144+0000 0 1 periodic scrub scheduled @ 2024-03-31T12:14:45.338144+0000 20 0 2024-03-30T12:14:50.971 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T12:14:50.971 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T12:14:50.971 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T12:14:50.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-03-30T12:14:50.982 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:14:50.982 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:14:51.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:14:51.287 INFO:tasks.workunit.client.0.smithi003.stderr:1' 2024-03-30T12:14:51.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:14:51.287 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:14:51.288 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:14:51.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247814 2024-03-30T12:14:51.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247814 2024-03-30T12:14:51.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814' 2024-03-30T12:14:51.411 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:14:51.411 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:14:51.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-03-30T12:14:51.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-03-30T12:14:51.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814 1-85899345927' 2024-03-30T12:14:51.516 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:51.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247814 2024-03-30T12:14:51.517 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:51.517 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:14:51.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247814 2024-03-30T12:14:51.518 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:51.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247814 2024-03-30T12:14:51.519 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 98784247814 2024-03-30T12:14:51.520 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2024-03-30T12:14:51.520 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:51.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247814 2024-03-30T12:14:51.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:14:52.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:14:52.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:53.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247814 2024-03-30T12:14:53.101 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:14:54.102 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-30T12:14:54.103 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:14:54.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247814 -lt 98784247814 2024-03-30T12:14:54.394 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:14:54.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345927 2024-03-30T12:14:54.395 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:14:54.396 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:14:54.396 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345927 2024-03-30T12:14:54.397 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:14:54.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-03-30T12:14:54.398 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345927' 2024-03-30T12:14:54.398 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 85899345927 2024-03-30T12:14:54.398 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:14:54.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345927 2024-03-30T12:14:54.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-03-30T12:14:54.678 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-03-30T12:14:54.785 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_bytes": 210, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects": 30, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_object_clones": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_object_copies": 60, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_missing_on_primary": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_missing": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_degraded": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_misplaced": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_unfound": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_dirty": 30, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_whiteouts": 0, 2024-03-30T12:14:54.786 INFO:tasks.workunit.client.0.smithi003.stdout: "num_read": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_read_kb": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_write": 30, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_write_kb": 30, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_scrub_errors": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_shallow_scrub_errors": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_deep_scrub_errors": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_recovered": 20, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_bytes_recovered": 140, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_keys_recovered": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_omap": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_hit_set_archive": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_bytes_hit_set_archive": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush_kb": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict": 0, 2024-03-30T12:14:54.787 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict_kb": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_promote": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush_mode_high": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush_mode_low": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict_mode_some": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict_mode_full": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_pinned": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_legacy_snapsets": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_large_omap_objects": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_manifest": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_omap_bytes": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_omap_keys": 0, 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_repaired": 20 2024-03-30T12:14:54.788 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:14:54.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-03-30T12:14:54.789 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-03-30T12:14:54.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-03-30T12:14:54.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-03-30T12:14:54.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-30T12:14:54.895 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-03-30T12:14:55.161 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:14:55.172 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:14:55.172 INFO:tasks.workunit.client.0.smithi003.stdout: "osd": 1, 2024-03-30T12:14:55.172 INFO:tasks.workunit.client.0.smithi003.stdout: "up_from": 20, 2024-03-30T12:14:55.172 INFO:tasks.workunit.client.0.smithi003.stdout: "seq": 85899345927, 2024-03-30T12:14:55.172 INFO:tasks.workunit.client.0.smithi003.stdout: "num_pgs": 1, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "num_osds": 1, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_osds": 1, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_omap_osds": 1, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "kb": 104857600, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used": 28552, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_data": 320, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_omap": 4, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_meta": 28219, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_avail": 104829048, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "statfs": { 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "total": 107374182400, 2024-03-30T12:14:55.173 INFO:tasks.workunit.client.0.smithi003.stdout: "available": 107344945152, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "internally_reserved": 0, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "allocated": 327680, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "data_stored": 53611, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed": 0, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_allocated": 0, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_original": 0, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "omap_allocated": 4398, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "internal_metadata": 28896978 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "hb_peers": [ 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: 0 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "snap_trim_queue_len": 0, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "num_snap_trimming": 0, 2024-03-30T12:14:55.174 INFO:tasks.workunit.client.0.smithi003.stdout: "num_shards_repaired": 10, 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "op_queue_age_hist": { 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "histogram": [], 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "upper_bound": 1 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "perf_stat": { 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ms": 0, 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ms": 0, 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ns": 0, 2024-03-30T12:14:55.175 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ns": 0 2024-03-30T12:14:55.176 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:55.176 INFO:tasks.workunit.client.0.smithi003.stdout: "alerts": [] 2024-03-30T12:14:55.176 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:14:55.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-30T12:14:55.177 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-03-30T12:14:55.438 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:14:55.449 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-03-30T12:14:55.449 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-03-30T12:14:55.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-03-30T12:14:55.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-30T12:14:55.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-30T12:14:55.715 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:14:55.726 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:14:55.726 INFO:tasks.workunit.client.0.smithi003.stdout: "osd": 0, 2024-03-30T12:14:55.726 INFO:tasks.workunit.client.0.smithi003.stdout: "up_from": 23, 2024-03-30T12:14:55.726 INFO:tasks.workunit.client.0.smithi003.stdout: "seq": 98784247815, 2024-03-30T12:14:55.726 INFO:tasks.workunit.client.0.smithi003.stdout: "num_pgs": 1, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "num_osds": 1, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_osds": 1, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_omap_osds": 1, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "kb": 104857600, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used": 28552, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_data": 320, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_omap": 4, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_meta": 28219, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_avail": 104829048, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "statfs": { 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "total": 107374182400, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "available": 107344945152, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "internally_reserved": 0, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "allocated": 327680, 2024-03-30T12:14:55.727 INFO:tasks.workunit.client.0.smithi003.stdout: "data_stored": 53611, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed": 0, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_allocated": 0, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_original": 0, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "omap_allocated": 4401, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "internal_metadata": 28896975 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "hb_peers": [ 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: 1 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "snap_trim_queue_len": 0, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "num_snap_trimming": 0, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "num_shards_repaired": 10, 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "op_queue_age_hist": { 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "histogram": [], 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: "upper_bound": 1 2024-03-30T12:14:55.728 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: "perf_stat": { 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ms": 0, 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ms": 0, 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ns": 0, 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ns": 0 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout: "alerts": [] 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:14:55.729 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-30T12:14:55.730 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-30T12:14:55.994 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:14:56.005 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-03-30T12:14:56.006 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-03-30T12:14:56.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-03-30T12:14:56.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-30T12:14:56.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-03-30T12:14:56.276 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:14:56.286 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:14:56.286 INFO:tasks.workunit.client.0.smithi003.stdout: "up_from": 0, 2024-03-30T12:14:56.286 INFO:tasks.workunit.client.0.smithi003.stdout: "seq": 0, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "num_pgs": 2, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "num_osds": 2, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_osds": 2, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_omap_osds": 2, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "kb": 209715200, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used": 57104, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_data": 640, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_omap": 8, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_meta": 56439, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_avail": 209658096, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "statfs": { 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "total": 214748364800, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "available": 214689890304, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "internally_reserved": 0, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "allocated": 655360, 2024-03-30T12:14:56.287 INFO:tasks.workunit.client.0.smithi003.stdout: "data_stored": 107222, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed": 0, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_allocated": 0, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_original": 0, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "omap_allocated": 8799, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "internal_metadata": 57793953 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "hb_peers": [], 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "snap_trim_queue_len": 0, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "num_snap_trimming": 0, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "num_shards_repaired": 20, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "op_queue_age_hist": { 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "histogram": [], 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "upper_bound": 1 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "perf_stat": { 2024-03-30T12:14:56.288 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ms": 0, 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ms": 0, 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ns": 0, 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ns": 0 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout: "alerts": [], 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout: "network_ping_times": [] 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-30T12:14:56.289 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-03-30T12:14:56.555 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:14:56.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:14:56.676 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:14:56.676 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:14:56.677 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:14:56.677 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:14:56.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:14:56.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:14:56.679 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:14:56.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:14:56.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:14:56.680 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:14:56.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:14:56.681 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:14:56.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:14:56.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:14:56.694 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:14:56.694 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:56.694 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:56.694 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-30T12:14:56.696 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:14:56.697 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:14:56.698 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:14:56.699 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:14:56.699 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:14:56.700 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:14:56.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:14:56.701 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:14:56.701 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:14:56.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:14:56.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:14:56.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:14:56.703 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:14:56.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:14:56.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-30T12:14:56.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:14:56.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:14:56.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:56.706 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:56.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:14:56.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:14:56.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:14:56.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-30T12:14:56.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-30T12:14:56.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:56.708 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:56.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69973 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-03-30T12:14:56.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-30T12:14:56.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:57.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-30T12:14:57.279 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-30T12:14:57.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-30T12:14:57.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:14:57.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:14:57.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-30T12:14:57.280 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:57.281 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:14:57.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:14:57.282 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get fsid 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-30T12:14:57.368 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-30T12:14:57.369 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-30T12:14:57.369 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-30T12:14:57.369 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:57.370 INFO:tasks.workunit.client.0.smithi003.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:57.370 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69973/ceph-mon.a.asok 2024-03-30T12:14:57.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-30T12:14:57.370 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69973/ceph-mon.a.asok config get mon_host 2024-03-30T12:14:57.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-03-30T12:14:57.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-30T12:14:57.450 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-30T12:14:57.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-30T12:14:57.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-30T12:14:57.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-30T12:14:57.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-30T12:14:58.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-30T12:14:58.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:14:58.266 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:14:58.266 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:14:58.266 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:58.267 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:58.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:58.267 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:14:58.267 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-30T12:14:58.294 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:14:58.295 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-03-30T12:14:58.296 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:14:58.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:14:58.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:14:58.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:14:58.303 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:14:58.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39348356-5155-4ba6-995e-24cae172d76a 2024-03-30T12:14:58.303 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 39348356-5155-4ba6-995e-24cae172d76a' 2024-03-30T12:14:58.303 INFO:tasks.workunit.client.0.smithi003.stdout:add osd0 39348356-5155-4ba6-995e-24cae172d76a 2024-03-30T12:14:58.304 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:14:58.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCAghmjsOKExAAKXqRV1gONSHUYO2NnVTQCw== 2024-03-30T12:14:58.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCAghmjsOKExAAKXqRV1gONSHUYO2NnVTQCw=="}' 2024-03-30T12:14:58.320 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39348356-5155-4ba6-995e-24cae172d76a -i td/osd-scrub-repair/0/new.json 2024-03-30T12:14:58.561 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:14:58.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-30T12:14:58.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBCAghmjsOKExAAKXqRV1gONSHUYO2NnVTQCw== --osd-uuid 39348356-5155-4ba6-995e-24cae172d76a 2024-03-30T12:14:58.602 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.610+0000 7f513680c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:58.612 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.620+0000 7f513680c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:58.613 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.622+0000 7f513680c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:14:58.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.622+0000 7f513680c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:14:58.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.622+0000 7f513680c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:14:58.614 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.622+0000 7f513680c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-30T12:14:58.704 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:14:58.712+0000 7f513680c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-30T12:15:04.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-30T12:15:04.361 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:15:04.362 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd0 key to auth repository 2024-03-30T12:15:04.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-30T12:15:04.362 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:15:04.734 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:15:04.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-30T12:15:04.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:15:04.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:15:04.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:15:04.735 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:15:04.762 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:04.770+0000 7f36c8fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:04.763 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:04.772+0000 7f36c8fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:04.764 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:04.773+0000 7f36c8fac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:15:05.017 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:15:05.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:05.018 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:05.281 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:05.289+0000 7f36c8fac640 -1 Falling back to public interface 2024-03-30T12:15:05.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:06.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:06.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:06.301 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:15:06.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:15:06.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:06.302 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:06.590 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:07.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:07.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:07.592 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:15:07.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:15:07.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:07.593 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:07.627 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:07.635+0000 7f36c8fac640 -1 osd.0 0 log_to_monitors true 2024-03-30T12:15:08.061 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:09.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:09.063 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:09.063 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:15:09.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:15:09.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:09.064 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:09.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:10.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:10.367 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:10.368 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:15:10.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:15:10.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:10.368 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:10.661 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:11.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:11.663 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:11.663 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:15:11.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:15:11.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:11.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:11.956 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:12.957 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:12.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:12.958 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:15:12.958 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:15:12.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:12.959 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:13.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:14.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:14.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:14.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:15:14.255 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:15:14.255 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:14.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:14.584 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:15.585 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:15.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:15.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:15:15.586 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:15:15.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:15.586 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:15.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:16.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:16.883 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:16.883 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:15:16.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:15:16.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:16.884 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:17.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:18.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:18.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:18.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:15:18.177 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:15:18.177 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:18.178 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:18.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:19.567 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:19.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:19.568 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:15:19.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:15:19.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:19.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:19.915 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:20.916 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:20.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:20.917 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:15:20.917 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:15:20.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:20.918 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:21.451 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:22.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:22.452 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:22.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:15:22.453 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:15:22.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:22.453 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:23.232 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:24.233 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:24.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:24.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:15:24.234 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:15:24.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:24.234 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:24.744 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:25.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:25.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:25.745 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:15:25.745 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:15:25.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:25.746 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:26.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:27.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:15:27.406 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:15:27.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:27.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:28.148 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:29.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:29.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:29.150 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:15:29.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:15:29.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:29.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:29.570 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:30.571 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:30.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:30.572 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:15:30.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:15:30.572 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:30.573 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:15:30.864 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2197064546,v1:127.0.0.1:6803/2197064546] [v2:127.0.0.1:6804/2197064546,v1:127.0.0.1:6805/2197064546] exists,up 39348356-5155-4ba6-995e-24cae172d76a 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:15:30.868 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:15:30.869 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:15:30.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=54b35ad9-9fd6-404a-a19a-7e893344de4b 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stdout:add osd1 54b35ad9-9fd6-404a-a19a-7e893344de4b 2024-03-30T12:15:30.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 54b35ad9-9fd6-404a-a19a-7e893344de4b' 2024-03-30T12:15:30.872 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:15:30.885 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiAghmBEZHNRAAwm94dONc3kLJ/dOKwQ85Rw== 2024-03-30T12:15:30.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiAghmBEZHNRAAwm94dONc3kLJ/dOKwQ85Rw=="}' 2024-03-30T12:15:30.886 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 54b35ad9-9fd6-404a-a19a-7e893344de4b -i td/osd-scrub-repair/1/new.json 2024-03-30T12:15:31.237 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:15:31.247 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-30T12:15:31.248 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBiAghmBEZHNRAAwm94dONc3kLJ/dOKwQ85Rw== --osd-uuid 54b35ad9-9fd6-404a-a19a-7e893344de4b 2024-03-30T12:15:31.277 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.285+0000 7fe9ebe60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:31.320 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.328+0000 7fe9ebe60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:31.321 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.329+0000 7fe9ebe60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.330+0000 7fe9ebe60640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:15:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.330+0000 7fe9ebe60640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:15:31.322 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.330+0000 7fe9ebe60640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-30T12:15:31.390 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:31.399+0000 7fe9ebe60640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-30T12:15:37.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-30T12:15:37.150 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:15:37.150 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd1 key to auth repository 2024-03-30T12:15:37.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-30T12:15:37.151 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:15:37.518 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:15:37.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-30T12:15:37.518 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:15:37.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:15:37.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:15:37.519 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:15:37.545 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:37.553+0000 7fad4c78c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:37.547 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:37.555+0000 7fad4c78c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:37.548 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:37.556+0000 7fad4c78c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:37.804 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:15:37.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:15:37.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:37.805 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:38.066 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:38.074+0000 7fad4c78c640 -1 Falling back to public interface 2024-03-30T12:15:38.086 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:39.088 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:39.088 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:15:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:15:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:39.089 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:39.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:40.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:40.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:40.390 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:15:40.390 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:15:40.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:40.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:40.393 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:15:40.401+0000 7fad4c78c640 -1 osd.1 0 log_to_monitors true 2024-03-30T12:15:40.709 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:41.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:41.711 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:41.711 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:15:41.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:15:41.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:41.712 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:42.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:43.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:43.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:43.008 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:15:43.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:15:43.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:43.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:43.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:44.439 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:15:44.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:15:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:44.440 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:44.821 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:45.822 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:45.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:45.823 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:15:45.823 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:15:45.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:45.824 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:46.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:47.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:47.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:47.193 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:15:47.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:15:47.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:47.194 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:47.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:48.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:48.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:48.486 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:15:48.486 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:15:48.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:48.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:48.775 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:49.776 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:49.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:49.777 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:15:49.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:15:49.777 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:49.778 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:50.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:51.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:51.200 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:51.201 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:15:51.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:15:51.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:51.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:51.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:52.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:52.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:52.618 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:15:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:15:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:52.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:52.910 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:53.912 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:53.912 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:15:53.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:53.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:15:53.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:53.913 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:54.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:55.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:55.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:55.422 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:15:55.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:15:55.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:55.423 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:56.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:57.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:57.111 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:15:57.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:15:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:57.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:57.703 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:15:58.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:15:58.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:15:58.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:15:58.705 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:15:58.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:15:58.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:15:59.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:00.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:00.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:00.424 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:16:00.425 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:16:00.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:00.425 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:16:01.174 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:02.175 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:02.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:02.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:16:02.176 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:16:02.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:02.176 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:16:03.011 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:04.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:04.012 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:04.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:16:04.013 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:16:04.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:04.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:16:04.307 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1949919896,v1:127.0.0.1:6811/1949919896] [v2:127.0.0.1:6812/1949919896,v1:127.0.0.1:6813/1949919896] exists,up 54b35ad9-9fd6-404a-a19a-7e893344de4b 2024-03-30T12:16:04.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:16:04.311 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-30T12:16:04.312 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:16:04.313 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36a26e7a-1507-404d-bd56-5854aeb4d24e 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 36a26e7a-1507-404d-bd56-5854aeb4d24e' 2024-03-30T12:16:04.314 INFO:tasks.workunit.client.0.smithi003.stdout:add osd2 36a26e7a-1507-404d-bd56-5854aeb4d24e 2024-03-30T12:16:04.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-30T12:16:04.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEAghmxgwdFBAAkv6fQur+KkdSoZw2LMTwjw== 2024-03-30T12:16:04.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEAghmxgwdFBAAkv6fQur+KkdSoZw2LMTwjw=="}' 2024-03-30T12:16:04.329 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36a26e7a-1507-404d-bd56-5854aeb4d24e -i td/osd-scrub-repair/2/new.json 2024-03-30T12:16:04.712 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:16:04.723 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-30T12:16:04.724 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCEAghmxgwdFBAAkv6fQur+KkdSoZw2LMTwjw== --osd-uuid 36a26e7a-1507-404d-bd56-5854aeb4d24e 2024-03-30T12:16:04.754 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.762+0000 7f8c69414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:16:04.789 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.797+0000 7f8c69414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:16:04.790 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.798+0000 7f8c69414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:16:04.790 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.798+0000 7f8c69414640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:16:04.790 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.798+0000 7f8c69414640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:16:04.790 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.798+0000 7f8c69414640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-30T12:16:04.863 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:04.871+0000 7f8c69414640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-30T12:16:10.810 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-30T12:16:10.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-30T12:16:10.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-30T12:16:10.812 INFO:tasks.workunit.client.0.smithi003.stdout:adding osd2 key to auth repository 2024-03-30T12:16:10.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-30T12:16:11.183 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.2 2024-03-30T12:16:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-30T12:16:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:16:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-30T12:16:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-30T12:16:11.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-30T12:16:11.213 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:11.219+0000 7fb7fbe0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:16:11.213 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:11.221+0000 7fb7fbe0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:16:11.214 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:11.222+0000 7fb7fbe0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:11.483 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:16:11.484 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:16:11.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:11.484 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:11.741 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:11.749+0000 7fb7fbe0c640 -1 Falling back to public interface 2024-03-30T12:16:11.763 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:12.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:12.765 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:16:12.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:16:12.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:12.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:13.054 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:14.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:14.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:14.056 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:16:14.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:16:14.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:14.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:14.079 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:16:14.087+0000 7fb7fbe0c640 -1 osd.2 0 log_to_monitors true 2024-03-30T12:16:14.386 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:15.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:15.388 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:15.388 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:16:15.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:16:15.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:15.389 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:15.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:16.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:16.684 INFO:tasks.workunit.client.0.smithi003.stdout:4 2024-03-30T12:16:16.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:16.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-30T12:16:16.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:16.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:17.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:18.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:18.048 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:18.049 INFO:tasks.workunit.client.0.smithi003.stdout:5 2024-03-30T12:16:18.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-30T12:16:18.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:18.049 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:18.341 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:19.342 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:19.343 INFO:tasks.workunit.client.0.smithi003.stdout:6 2024-03-30T12:16:19.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:19.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-30T12:16:19.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:19.343 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:19.656 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:20.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:20.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:20.657 INFO:tasks.workunit.client.0.smithi003.stdout:7 2024-03-30T12:16:20.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-30T12:16:20.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:20.658 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:20.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:21.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:21.954 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:21.955 INFO:tasks.workunit.client.0.smithi003.stdout:8 2024-03-30T12:16:21.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-30T12:16:21.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:21.955 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:22.251 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:23.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:23.253 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:23.253 INFO:tasks.workunit.client.0.smithi003.stdout:9 2024-03-30T12:16:23.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-30T12:16:23.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:23.254 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:23.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:24.618 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:24.619 INFO:tasks.workunit.client.0.smithi003.stdout:10 2024-03-30T12:16:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-30T12:16:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:24.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:24.927 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:25.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:25.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:25.929 INFO:tasks.workunit.client.0.smithi003.stdout:11 2024-03-30T12:16:25.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-30T12:16:25.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:25.930 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:26.223 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:27.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:27.225 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:27.225 INFO:tasks.workunit.client.0.smithi003.stdout:12 2024-03-30T12:16:27.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-30T12:16:27.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:27.226 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:27.706 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:28.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:28.707 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:28.707 INFO:tasks.workunit.client.0.smithi003.stdout:13 2024-03-30T12:16:28.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-30T12:16:28.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:28.708 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:29.106 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:30.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:30.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:30.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-30T12:16:30.108 INFO:tasks.workunit.client.0.smithi003.stdout:14 2024-03-30T12:16:30.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:30.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:31.044 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:32.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:32.046 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:32.046 INFO:tasks.workunit.client.0.smithi003.stdout:15 2024-03-30T12:16:32.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-30T12:16:32.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:32.047 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:32.702 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:33.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:33.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:33.704 INFO:tasks.workunit.client.0.smithi003.stdout:16 2024-03-30T12:16:33.704 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-30T12:16:33.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:33.705 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:34.182 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:35.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:35.183 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:35.183 INFO:tasks.workunit.client.0.smithi003.stdout:17 2024-03-30T12:16:35.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-30T12:16:35.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:35.184 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:35.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:16:36.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:16:36.541 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:16:36.542 INFO:tasks.workunit.client.0.smithi003.stdout:18 2024-03-30T12:16:36.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-30T12:16:36.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:16:36.542 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-30T12:16:36.837 INFO:tasks.workunit.client.0.smithi003.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3032988715,v1:127.0.0.1:6819/3032988715] [v2:127.0.0.1:6820/3032988715,v1:127.0.0.1:6821/3032988715] exists,up 36a26e7a-1507-404d-bd56-5854aeb4d24e 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-30T12:16:36.838 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-30T12:16:36.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-30T12:16:36.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-30T12:16:37.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-03-30T12:16:37.288 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-03-30T12:16:38.021 INFO:tasks.workunit.client.0.smithi003.stderr:pool 'testpool' created 2024-03-30T12:16:38.033 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-30T12:16:39.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-30T12:16:39.035 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-30T12:16:39.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:16:39.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:16:39.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:16:39.036 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:16:39.037 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:16:39.037 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:16:39.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:16:39.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:16:39.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:16:39.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:16:39.137 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:16:39.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:16:39.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:16:39.138 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:16:39.138 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:16:39.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:16:39.464 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:16:39.464 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:16:39.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:16:39.464 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:16:39.464 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:16:39.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-30T12:16:39.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-30T12:16:39.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-30T12:16:39.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:16:39.579 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:16:39.764 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-30T12:16:39.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-30T12:16:39.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-03-30T12:16:39.765 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:16:39.765 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:16:39.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-30T12:16:39.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-30T12:16:39.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509447' 2024-03-30T12:16:39.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:16:39.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-30T12:16:39.873 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:16:39.873 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:16:39.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-30T12:16:39.874 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:16:39.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-30T12:16:39.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-30T12:16:39.876 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 21474836498 2024-03-30T12:16:39.876 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:16:40.158 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-03-30T12:16:40.159 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:16:41.160 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:16:41.160 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:16:41.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-03-30T12:16:41.485 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:16:41.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-30T12:16:41.486 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:16:41.487 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:16:41.487 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-30T12:16:41.488 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:16:41.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-30T12:16:41.489 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-30T12:16:41.489 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 42949672972 2024-03-30T12:16:41.489 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:16:41.770 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-30T12:16:41.771 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:16:41.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-30T12:16:41.771 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:16:41.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:16:41.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-30T12:16:41.773 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:16:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-30T12:16:41.774 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-30T12:16:41.774 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509447 2024-03-30T12:16:41.775 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:16:42.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-30T12:16:42.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:16:42.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:16:42.056 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:16:42.407 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:16:42.684 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T12:16:42.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:16:42.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:16:42.685 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-03-30T12:16:43.037 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-03-30T12:16:43.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:43.038 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:43.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:43.186 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:43.404 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:43.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:43.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:43.631 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:43.807 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:43.808 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:44.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:44.001 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:44.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:44.235 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:44.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:44.445 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:44.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:44.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:44.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:44.881 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:45.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:45.108 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:45.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:45.292 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:45.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:45.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:45.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:45.754 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:45.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:45.980 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:46.197 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:46.198 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:46.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:46.382 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:46.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:46.592 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:46.759 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:46.760 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:47.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:47.003 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:47.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:47.195 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:47.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:47.422 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:47.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:47.657 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:47.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:47.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:48.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:48.085 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:48.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:48.269 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:48.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:48.512 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:48.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:48.755 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:48.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:48.965 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:49.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-30T12:16:49.166 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T12:16:49.352 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T12:16:49.353 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T12:16:49.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-30T12:16:49.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-30T12:16:49.649 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-30T12:16:49.932 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-03-30T12:16:49.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-03-30T12:16:49.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-30T12:16:49.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-30T12:16:49.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-30T12:16:49.933 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-30T12:16:50.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-03-30T12:16:50.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-03-30T12:16:50.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-30T12:16:50.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-30T12:16:50.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-30T12:16:50.219 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:16:50.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:16:50.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:16:50.818 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-30T12:16:50.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:16:50.819 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:16:50.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:16:50.819 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:16:50.820 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:16:51.126 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:16:51.126 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-03-30T12:16:51.127 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:51.127 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:16:51.129 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-30T12:16:52.424 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-03-30T12:16:52.939 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:52.939 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-03-30T12:16:52.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:16:52.941 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-30T12:16:52.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:52.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:52.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:16:52.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:52.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:16:52.942 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-30T12:16:54.091 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:104778fc:::obj2:head# 2024-03-30T12:16:54.606 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:54.607 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-03-30T12:16:54.608 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:16:54.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-30T12:16:55.432 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-03-30T12:16:55.949 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:55.949 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-03-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:55.951 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:16:55.952 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-30T12:16:56.752 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-03-30T12:16:57.271 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:57.271 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-03-30T12:16:57.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:16:57.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-30T12:16:57.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:57.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:57.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:16:57.273 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:57.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:16:57.274 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-30T12:16:57.979 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:c52c9666:::obj5:head# 2024-03-30T12:16:58.501 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:58.501 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-03-30T12:16:58.503 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:16:58.504 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-30T12:16:59.165 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:233a42c1:::obj6:head# 2024-03-30T12:16:59.679 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:16:59.680 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-03-30T12:16:59.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:16:59.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-30T12:16:59.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:16:59.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:16:59.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:16:59.682 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:16:59.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:16:59.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-30T12:17:00.325 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:9a09113a:::obj7:head# 2024-03-30T12:17:00.840 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:00.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:00.843 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-30T12:17:01.602 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:a557efb1:::obj8:head# 2024-03-30T12:17:02.119 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:02.119 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-03-30T12:17:02.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:02.121 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-30T12:17:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:02.122 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-30T12:17:02.954 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:fe351e27:::obj9:head# 2024-03-30T12:17:03.473 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:03.473 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-03-30T12:17:03.475 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:03.476 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-30T12:17:04.115 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:d1337354:::obj10:head# 2024-03-30T12:17:04.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:04.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-03-30T12:17:04.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:04.639 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-30T12:17:04.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:04.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:04.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:04.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:04.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:04.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-30T12:17:05.266 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-03-30T12:17:06.034 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:06.035 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-03-30T12:17:06.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:06.036 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-30T12:17:06.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:06.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:06.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:06.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:06.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:06.037 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-30T12:17:06.703 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-03-30T12:17:07.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:07.470 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:07.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-30T12:17:08.048 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-03-30T12:17:08.566 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:08.566 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:08.568 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-30T12:17:09.200 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:e0a44829:::obj14:head# 2024-03-30T12:17:09.718 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:09.718 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-03-30T12:17:09.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:09.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-30T12:17:09.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:09.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:09.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:09.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:09.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:09.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-30T12:17:10.310 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:ab946124:::obj15:head# 2024-03-30T12:17:10.831 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:10.831 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-03-30T12:17:10.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:10.833 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-30T12:17:10.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:10.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:10.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:10.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:10.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:10.834 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-30T12:17:11.494 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-03-30T12:17:12.013 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:12.013 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:12.015 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-30T12:17:12.689 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-03-30T12:17:13.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:13.203 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-03-30T12:17:13.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:13.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-30T12:17:13.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:13.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:13.205 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:13.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:13.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:13.206 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-30T12:17:13.832 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:b9836a99:::obj18:head# 2024-03-30T12:17:14.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:14.350 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-03-30T12:17:14.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:14.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-30T12:17:14.352 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:14.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:14.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:14.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:14.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:14.353 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-30T12:17:14.950 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-03-30T12:17:15.469 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:15.469 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-03-30T12:17:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-30T12:17:15.471 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:15.472 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-30T12:17:16.085 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:24b5611b:::obj20:head# 2024-03-30T12:17:16.609 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:16.609 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-03-30T12:17:16.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:16.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-03-30T12:17:16.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:16.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:16.611 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:16.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:16.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:16.612 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-03-30T12:17:17.280 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-03-30T12:17:17.798 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:17.798 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-03-30T12:17:17.802 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:17.803 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-03-30T12:17:18.456 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:c0c49def:::obj22:head# 2024-03-30T12:17:18.971 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:18.971 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-03-30T12:17:18.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:18.973 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-03-30T12:17:18.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:18.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:18.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:18.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:18.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:18.974 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-03-30T12:17:19.833 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:242c40e0:::obj23:head# 2024-03-30T12:17:20.348 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:20.349 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-03-30T12:17:20.350 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:20.351 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-03-30T12:17:21.010 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:7eacede5:::obj24:head# 2024-03-30T12:17:21.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:21.528 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-03-30T12:17:21.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-30T12:17:21.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-03-30T12:17:21.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:21.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:21.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-30T12:17:21.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:21.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:21.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-03-30T12:17:22.204 INFO:tasks.workunit.client.0.smithi003.stdout:remove 0#1:7109478f:::obj25:head# 2024-03-30T12:17:22.719 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-30T12:17:22.719 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-03-30T12:17:22.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-30T12:17:22.721 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-03-30T12:17:22.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-30T12:17:22.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-30T12:17:22.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-30T12:17:22.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-30T12:17:22.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:22.722 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-03-30T12:17:23.488 INFO:tasks.workunit.client.0.smithi003.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:17:24.006 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:17:24.007 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-30T12:17:24.008 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.1 2024-03-30T12:17:24.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-30T12:17:24.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:17:24.009 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-30T12:17:24.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-30T12:17:24.009 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:17:24.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:17:24.010 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:17:24.037 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:24.045+0000 7f1c51c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:17:24.040 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:24.048+0000 7f1c51c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:17:24.041 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:24.049+0000 7f1c51c10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:17:24.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-30T12:17:24.298 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:17:24.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-30T12:17:24.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:17:24.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:17:24.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:24.299 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:17:24.299 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:17:24.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:24.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:17:24.556 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:24.564+0000 7f1c51c10640 -1 Falling back to public interface 2024-03-30T12:17:24.578 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:17:25.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:17:25.579 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:25.580 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:17:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:17:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:25.580 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:17:25.874 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:17:26.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:17:26.875 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:26.875 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:17:26.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:17:26.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:26.876 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:17:26.876 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:26.885+0000 7f1c51c10640 -1 osd.1 22 log_to_monitors true 2024-03-30T12:17:27.191 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:17:28.192 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:17:28.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:28.193 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:17:28.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:17:28.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:28.193 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-30T12:17:28.486 INFO:tasks.workunit.client.0.smithi003.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/1470953009,v1:127.0.0.1:6803/1470953009] [v2:127.0.0.1:6804/1470953009,v1:127.0.0.1:6805/1470953009] exists,up 54b35ad9-9fd6-404a-a19a-7e893344de4b 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-30T12:17:28.490 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' 2024-03-30T12:17:28.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-30T12:17:28.492 INFO:tasks.workunit.client.0.smithi003.stdout:start osd.0 2024-03-30T12:17:28.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=32a4ec17-b7f8-4d07-9ab2-274f19304f60 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69973/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-30T12:17:28.493 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-30T12:17:28.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-30T12:17:28.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-30T12:17:28.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-30T12:17:28.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-30T12:17:28.516 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:28.525+0000 7f92cbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:17:28.518 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:28.527+0000 7f92cbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:17:28.519 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:28.528+0000 7f92cbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:17:28.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-30T12:17:28.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-30T12:17:28.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-30T12:17:28.772 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-30T12:17:28.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-30T12:17:28.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:28.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-30T12:17:28.773 INFO:tasks.workunit.client.0.smithi003.stdout:0 2024-03-30T12:17:28.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:28.773 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:17:29.035 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:29.043+0000 7f92cbc0d640 -1 Falling back to public interface 2024-03-30T12:17:29.053 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:17:30.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:17:30.055 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:30.055 INFO:tasks.workunit.client.0.smithi003.stdout:1 2024-03-30T12:17:30.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-30T12:17:30.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:30.056 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:17:30.345 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:17:31.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:17:31.346 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:31.347 INFO:tasks.workunit.client.0.smithi003.stdout:2 2024-03-30T12:17:31.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-30T12:17:31.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:31.347 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:17:31.347 INFO:tasks.workunit.client.0.smithi003.stderr:2024-03-30T12:17:31.355+0000 7f92cbc0d640 -1 osd.0 21 log_to_monitors true 2024-03-30T12:17:31.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-30T12:17:32.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-30T12:17:32.643 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-30T12:17:32.643 INFO:tasks.workunit.client.0.smithi003.stdout:3 2024-03-30T12:17:32.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-30T12:17:32.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-30T12:17:32.644 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-30T12:17:32.951 INFO:tasks.workunit.client.0.smithi003.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/2094326143,v1:127.0.0.1:6811/2094326143] [v2:127.0.0.1:6812/2094326143,v1:127.0.0.1:6813/2094326143] exists,up 39348356-5155-4ba6-995e-24cae172d76a 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:17:32.953 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:17:33.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:17:33.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:17:33.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:17:33.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:17:33.042 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:17:33.042 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:17:33.321 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:17:33.322 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:17:33.322 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:17:33.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:17:33.322 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:33.322 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:17:33.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-30T12:17:33.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-30T12:17:33.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-30T12:17:33.426 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:33.426 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:17:33.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-03-30T12:17:33.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-03-30T12:17:33.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-03-30T12:17:33.531 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:33.532 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:17:33.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-30T12:17:33.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-30T12:17:33.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509459' 2024-03-30T12:17:33.637 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:33.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-30T12:17:33.637 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:33.638 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:17:33.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-30T12:17:33.639 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:33.640 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-30T12:17:33.640 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 124554051586 2024-03-30T12:17:33.641 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-30T12:17:33.641 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:17:33.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-30T12:17:33.922 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:17:34.923 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:17:34.924 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:17:35.218 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-30T12:17:35.219 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:35.220 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149699 2024-03-30T12:17:35.220 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:35.220 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:17:35.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149699 2024-03-30T12:17:35.221 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:35.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-03-30T12:17:35.222 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-03-30T12:17:35.222 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 111669149699 2024-03-30T12:17:35.222 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:17:35.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-03-30T12:17:35.521 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:35.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-30T12:17:35.521 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:35.522 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:17:35.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-30T12:17:35.523 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:35.524 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-30T12:17:35.525 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-30T12:17:35.525 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509459 2024-03-30T12:17:35.525 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:17:35.806 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-30T12:17:35.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:17:35.806 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:17:35.807 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:17:36.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T12:17:36.164 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:17:36.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:17:36.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:17:36.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:17:36.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:17:36.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:17:36.165 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:17:36.446 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T12:17:36.446 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:17:36.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:17:36.447 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:17:36.796 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:17:37.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:37.087 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-30T12:17:37.354 INFO:tasks.workunit.client.0.smithi003.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-03-30T12:17:37.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:37.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-30T12:17:37.365 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:17:37.366 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:17:37.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:16:37.913190+0000 '>' 2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:37.664 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:17:38.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:17:38.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:17:38.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:17:38.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:17:38.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:17:38.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:17:38.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:17:38.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:16:37.913190+0000 '>' 2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:38.960 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:17:39.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:17:39.961 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:17:39.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:17:39.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:17:39.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:17:39.962 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:17:39.963 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:17:40.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:16:37.913190+0000 '>' 2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:40.256 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:17:41.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:17:41.257 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:17:41.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:17:41.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:17:41.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:17:41.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:17:41.258 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:17:41.546 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:16:37.913190+0000 '>' 2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:41.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-30T12:17:42.547 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-30T12:17:42.548 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-30T12:17:42.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-30T12:17:42.548 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-30T12:17:42.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-30T12:17:42.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-30T12:17:42.549 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-30T12:17:42.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-30T12:17:38.006124+0000 '>' 2024-03-30T12:16:37.913190+0000 2024-03-30T12:17:42.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-30T12:17:42.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-03-30T12:17:42.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-30T12:17:42.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-30T12:17:42.839 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-30T12:17:42.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-30T12:17:42.840 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-30T12:17:42.840 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-30T12:17:42.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-30T12:17:42.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-30T12:17:42.840 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-30T12:17:42.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-30T12:17:42.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-30T12:17:42.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-30T12:17:42.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-30T12:17:42.929 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:17:42.929 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:17:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:17:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:17:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:17:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:17:43.210 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:43.210 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:17:43.314 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-30T12:17:43.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-30T12:17:43.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-30T12:17:43.315 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:43.315 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:17:43.420 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-30T12:17:43.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-30T12:17:43.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2024-03-30T12:17:43.421 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:43.421 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:17:43.526 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-30T12:17:43.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-30T12:17:43.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509462' 2024-03-30T12:17:43.527 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:43.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-30T12:17:43.527 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:43.528 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:17:43.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-30T12:17:43.529 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:43.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-30T12:17:43.530 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-30T12:17:43.531 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 124554051589 2024-03-30T12:17:43.531 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:17:43.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-30T12:17:43.811 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:17:44.812 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:17:44.812 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:17:45.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-03-30T12:17:45.107 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:45.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149702 2024-03-30T12:17:45.108 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:45.109 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:17:45.109 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149702 2024-03-30T12:17:45.110 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:45.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-30T12:17:45.111 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2024-03-30T12:17:45.111 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 111669149702 2024-03-30T12:17:45.111 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:17:45.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-30T12:17:45.435 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:45.435 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-30T12:17:45.436 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:45.437 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:17:45.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-30T12:17:45.437 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-30T12:17:45.439 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-30T12:17:45.439 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509462 2024-03-30T12:17:45.439 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:17:45.720 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-30T12:17:45.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-30T12:17:45.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:17:45.721 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:17:46.112 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-30T12:17:46.113 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-30T12:17:46.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-30T12:17:46.113 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-30T12:17:46.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-30T12:17:46.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-30T12:17:46.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-30T12:17:46.114 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-30T12:17:46.391 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-30T12:17:46.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-30T12:17:46.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-30T12:17:46.392 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-30T12:17:46.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-30T12:17:46.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-30T12:17:46.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-30T12:17:46.743 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-03-30T12:17:47.009 INFO:tasks.workunit.client.0.smithi003.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-30T12:17:47.009 INFO:tasks.workunit.client.0.smithi003.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-30T12:17:39.270451+0000 21'30 30:130 [1,0,2] 1 [1,0,2] 1 21'30 2024-03-30T12:17:38.006124+0000 21'30 2024-03-30T12:17:38.006124+0000 0 1 periodic scrub scheduled @ 2024-03-31T12:17:38.006124+0000 17 0 2024-03-30T12:17:47.009 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-03-30T12:17:47.009 INFO:tasks.workunit.client.0.smithi003.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-30T12:17:47.009 INFO:tasks.workunit.client.0.smithi003.stderr:dumped pgs 2024-03-30T12:17:47.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-03-30T12:17:47.020 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-30T12:17:47.020 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-30T12:17:47.300 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-30T12:17:47.300 INFO:tasks.workunit.client.0.smithi003.stderr:1 2024-03-30T12:17:47.301 INFO:tasks.workunit.client.0.smithi003.stderr:2' 2024-03-30T12:17:47.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-30T12:17:47.301 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:47.301 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-30T12:17:47.405 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-03-30T12:17:47.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-03-30T12:17:47.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-03-30T12:17:47.406 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:47.406 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-30T12:17:47.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149704 2024-03-30T12:17:47.510 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149704 2024-03-30T12:17:47.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-03-30T12:17:47.511 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-30T12:17:47.511 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-30T12:17:47.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-30T12:17:47.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-30T12:17:47.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509464' 2024-03-30T12:17:47.616 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:47.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-03-30T12:17:47.616 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:47.617 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-30T12:17:47.618 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-03-30T12:17:47.618 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:47.619 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-03-30T12:17:47.619 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.0 seq 124554051591 2024-03-30T12:17:47.620 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-03-30T12:17:47.620 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:17:47.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051591 2024-03-30T12:17:47.900 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-30T12:17:48.901 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-30T12:17:48.901 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-30T12:17:49.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-03-30T12:17:49.199 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:49.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149704 2024-03-30T12:17:49.200 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:49.201 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-30T12:17:49.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149704 2024-03-30T12:17:49.202 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:49.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149704 2024-03-30T12:17:49.203 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.1 seq 111669149704 2024-03-30T12:17:49.203 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-03-30T12:17:49.204 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-30T12:17:49.491 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149704 2024-03-30T12:17:49.492 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-30T12:17:49.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-30T12:17:49.492 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-30T12:17:49.493 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-30T12:17:49.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-30T12:17:49.494 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-30T12:17:49.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-30T12:17:49.495 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-30T12:17:49.496 INFO:tasks.workunit.client.0.smithi003.stdout:waiting osd.2 seq 64424509464 2024-03-30T12:17:49.496 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-30T12:17:49.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-30T12:17:49.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-30T12:17:49.781 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-03-30T12:17:49.888 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_bytes": 210, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects": 30, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_object_clones": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_object_copies": 90, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_missing_on_primary": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_missing": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_degraded": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_misplaced": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_unfound": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_dirty": 30, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_whiteouts": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_read": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_read_kb": 0, 2024-03-30T12:17:49.889 INFO:tasks.workunit.client.0.smithi003.stdout: "num_write": 30, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_write_kb": 30, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_scrub_errors": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_shallow_scrub_errors": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_deep_scrub_errors": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_recovered": 26, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_bytes_recovered": 182, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_keys_recovered": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_omap": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_hit_set_archive": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_bytes_hit_set_archive": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush_kb": 0, 2024-03-30T12:17:49.890 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict_kb": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_promote": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush_mode_high": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_flush_mode_low": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict_mode_some": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_evict_mode_full": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_pinned": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_legacy_snapsets": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_large_omap_objects": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_manifest": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_omap_bytes": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_omap_keys": 0, 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout: "num_objects_repaired": 26 2024-03-30T12:17:49.891 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:17:49.892 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-30T12:17:49.892 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-03-30T12:17:49.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-03-30T12:17:49.998 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-03-30T12:17:49.998 INFO:tasks.workunit.client.0.smithi003.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-03-30T12:17:49.999 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-03-30T12:17:50.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-30T12:17:50.000 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-03-30T12:17:50.000 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-03-30T12:17:50.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-03-30T12:17:50.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-30T12:17:50.008 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-30T12:17:50.270 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout:{ 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout: "osd": 0, 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout: "up_from": 29, 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout: "seq": 124554051591, 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout: "num_pgs": 1, 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout: "num_osds": 1, 2024-03-30T12:17:50.278 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_osds": 1, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "num_per_pool_omap_osds": 1, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "kb": 104857600, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used": 28472, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_data": 368, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_omap": 4, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_used_meta": 28091, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "kb_avail": 104829128, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "statfs": { 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "total": 107374182400, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "available": 107345027072, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "internally_reserved": 0, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "allocated": 376832, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "data_stored": 203264, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed": 0, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_allocated": 0, 2024-03-30T12:17:50.279 INFO:tasks.workunit.client.0.smithi003.stdout: "data_compressed_original": 0, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "omap_allocated": 4402, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "internal_metadata": 28765902 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "hb_peers": [ 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: 1, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: 2 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: ], 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "snap_trim_queue_len": 0, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "num_snap_trimming": 0, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "num_shards_repaired": 26, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "op_queue_age_hist": { 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "histogram": [], 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "upper_bound": 1 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "perf_stat": { 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ms": 0, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ms": 0, 2024-03-30T12:17:50.280 INFO:tasks.workunit.client.0.smithi003.stdout: "commit_latency_ns": 0, 2024-03-30T12:17:50.281 INFO:tasks.workunit.client.0.smithi003.stdout: "apply_latency_ns": 0 2024-03-30T12:17:50.281 INFO:tasks.workunit.client.0.smithi003.stdout: }, 2024-03-30T12:17:50.281 INFO:tasks.workunit.client.0.smithi003.stdout: "alerts": [] 2024-03-30T12:17:50.281 INFO:tasks.workunit.client.0.smithi003.stdout:} 2024-03-30T12:17:50.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-30T12:17:50.281 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-30T12:17:50.544 INFO:tasks.workunit.client.0.smithi003.stderr:dumped all 2024-03-30T12:17:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-03-30T12:17:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-03-30T12:17:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-03-30T12:17:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-03-30T12:17:50.556 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-30T12:17:50.557 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-30T12:17:50.665 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-30T12:17:50.666 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-30T12:17:50.666 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-30T12:17:50.667 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-30T12:17:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-30T12:17:50.668 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-30T12:17:50.668 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-30T12:17:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:17:50.669 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-30T12:17:50.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-30T12:17:50.670 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:17:50.670 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:17:50.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-03-30T12:17:50.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-03-30T12:17:50.672 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-03-30T12:17:50.673 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-03-30T12:17:50.674 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-30T12:17:50.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-30T12:17:50.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-30T12:17:50.683 INFO:tasks.workunit.client.0.smithi003.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69973 2024-03-30T12:17:50.683 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69973 2024-03-30T12:17:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:17:50.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-30T12:17:50.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-30T12:17:50.685 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-03-30T12:17:50.685 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-03-30T12:17:50.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T12:17:50.971 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-30T12:17:51.192 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4d9772a58cb2409e9ccaf8c865e0b437 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-30T12:17:51.195 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T12:17:51.204 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T12:17:51.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T12:17:51.256 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T12:17:51.265 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T12:17:51.265 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T12:17:51.290 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-30T12:17:51.294 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T12:17:51.294 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-30T12:17:51.294 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 76 -2008ns[-2008ns] +/- 83ms 2024-03-30T12:17:51.294 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 10 377 91 -354us[ -384us] +/- 62ms 2024-03-30T12:17:51.294 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 72 +422us[ +422us] +/- 75ms 2024-03-30T12:17:51.294 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T12:17:51.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T12:17:51.303 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T12:17:51.303 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T12:17:51.311 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T12:17:51.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T12:17:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:17:51.425 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T12:17:51.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T12:17:51.442 INFO:teuthology.task.internal:Duration was 5577.261185 seconds 2024-03-30T12:17:51.443 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T12:17:51.451 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T12:17:51.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T12:17:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T12:17:51.916 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T12:17:51.916 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-03-30T12:17:51.916 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T12:17:51.977 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T12:17:51.977 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T12:17:52.046 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T12:17:52.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T12:17:52.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T12:17:52.194 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T12:17:52.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T12:17:52.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T12:17:52.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:17:52.286 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-03-30T12:17:52.310 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T12:17:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:17:52.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T12:17:52.374 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T12:17:52.376 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473/remote/smithi003 2024-03-30T12:17:52.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T12:17:53.471 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T12:17:53.472 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T12:17:53.525 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T12:17:53.536 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T12:17:53.536 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T12:17:53.555 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T12:17:53.555 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T12:17:53.580 INFO:teuthology.orchestra.run.smithi003.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 12:17 /home/ubuntu/cephtest 2024-03-30T12:17:53.580 INFO:teuthology.orchestra.run.smithi003.stdout: 394760 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 10:51 /home/ubuntu/cephtest/mnt.0 2024-03-30T12:17:53.581 INFO:teuthology.orchestra.run.smithi003.stdout: 394761 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 11:01 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-30T12:17:53.581 INFO:teuthology.orchestra.run.smithi003.stdout: 394802 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 11:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-30T12:17:53.581 INFO:teuthology.orchestra.run.smithi003.stdout: 408961 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 12:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-03-30T12:17:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:17:53.582 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-30T12:17:53.582 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-30T12:17:53.587 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T12:17:53.597 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T12:17:53.607 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-30T12:17:53.607 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T12:17:53.630 DEBUG:teuthology.nuke:shortname: smithi003 2024-03-30T12:17:53.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T12:17:53.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630473', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 10:35:24.815388', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMsWYmf6Xl2fk/Utdo9pthxdi5pqHBqbjubEp7sYVvzAuoBQLg/OOaI4+ZCiLim6+jcEg8UegHmnG+KiyYyQAmQ='} 2024-03-30T12:17:53.700 INFO:teuthology.orchestra.console.smithi003:Power off 2024-03-30T12:17:53.700 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T12:17:53.723 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-03-30T12:17:53.723 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:17:53.746 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-30T12:17:57.746 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:17:57.856 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-30T12:18:01.861 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:18:01.970 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-30T12:18:05.974 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T12:18:06.085 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-03-30T12:18:06.185 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-03-30T12:18:06.291 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 5577.261184930801 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi003 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4d9772a58cb2409e9ccaf8c865e0b437 status: fail success: false 2024-03-30T12:18:06.291 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T12:18:06.370 INFO:teuthology.run:FAIL